ROIpad ← Back to Search
github.com › repository issue

w8a16 (int8 GPTQ) target: Marlin repack CUDA IMA (wedges card until reboot); every fallback route fails — Conch 23 tok/s, fp16 blocked by bf16-only spec-decode kernels

syv-ai/qwen38-27b-rtx3090
Status: Open
Opened: Aug 23, 2026
Comments: 7
## Environment - This repo @ `2ae239f` (vLLM 0.27.1 fork), launched via `single-user/start_qwen.sh` - GPU: **NVIDIA CMP 170HX 40GB** (GA100, sm80, compute capability 8.0), driver 610.43.03 - Target model: Qwen3.8-27B quantized to **GPTQ int8 (w8a16)**: `group_size=128, sym=True, desc_act=False`, config `dtype: float16`, fp16 lm_head/MTP, `--language-model-only` - Drafter: the shipped `Qwen3.8-27B-DFlash2-W4A16`, `SPEC=dflash2 CTX=fast` - Base command: `MODEL= SPEC=dflash2 CTX=fast GPU_UTIL=0.96 bash single-user/start_qwen.sh`, with the per-row kernel/dtype overrides below (`VLLM_DISABLED_KERNELS`, `EXTRA_ARGS`, `SPEC_ATTN=0`) I realize w8a16 targets aren't what this repo ships — but two of the failure modes below look like real bugs independent of which checkpoint triggers them, and the crash mode (card wedge until reboot) is nasty enough to be worth a guard at minimum. ## Result matrix (5 attempts, one card) | # | Linear kernel | `--dtype` | KV cache | Result | |---|---|---|---|---| | 1 | Marlin (default choice) | *(none → fp16)* | bfloat16 | **CUDA illegal memory access in `marlin_permute_scales` during `process_weights_after_loading`**; card wedged until reboot — see below | | 2 | Exllama (Marlin disabled) | *(none → fp16)* | bfloat16 | FA2 init: `mha_varlen_fwd ... query and key must have the same dtype` | | 3 | Conch (`conch-triton-kernels==1.3`) | bfloat16 | bfloat16 | **Boots and serves correctly — but 23 tok/s** (vs 133.7 tok/s for the shipped W4A...
Python
View on GitHub ↗