w8a16 (int8 GPTQ) target: Marlin repack CUDA IMA (wedges card until reboot); every fallback route fails — Conch 23 tok/s, fp16 blocked by bf16-only spec-decode kernels
syv-ai/qwen38-27b-rtx3090
## Environment
- This repo @ `2ae239f` (vLLM 0.27.1 fork), launched via `single-user/start_qwen.sh`
- GPU: **NVIDIA CMP 170HX 40GB** (GA100, sm80, compute capability 8.0), driver 610.43.03
- Target model: Qwen3.8-27B quantized to **GPTQ int8 (w8a16)**: `group_size=128, sym=True, desc_act=False`, config `dtype: float16`, fp16 lm_head/MTP, `--language-model-only`
- Drafter: the shipped `Qwen3.8-27B-DFlash2-W4A16`, `SPEC=dflash2 CTX=fast`
- Base command: `MODEL= SPEC=dflash2 CTX=fast GPU_UTIL=0.96 bash single-user/start_qwen.sh`, with the per-row kernel/dtype overrides below (`VLLM_DISABLED_KERNELS`, `EXTRA_ARGS`, `SPEC_ATTN=0`)
I realize w8a16 targets aren't what this repo ships — but two of the failure modes below look like real bugs independent of which checkpoint triggers them, and the crash mode (card wedge until reboot) is nasty enough to be worth a guard at minimum.
## Result matrix (5 attempts, one card)
| # | Linear kernel | `--dtype` | KV cache | Result |
|---|---|---|---|---|
| 1 | Marlin (default choice) | *(none → fp16)* | bfloat16 | **CUDA illegal memory access in `marlin_permute_scales` during `process_weights_after_loading`**; card wedged until reboot — see below |
| 2 | Exllama (Marlin disabled) | *(none → fp16)* | bfloat16 | FA2 init: `mha_varlen_fwd ... query and key must have the same dtype` |
| 3 | Conch (`conch-triton-kernels==1.3`) | bfloat16 | bfloat16 | **Boots and serves correctly — but 23 tok/s** (vs 133.7 tok/s for the shipped W4A...
View on GitHub ↗
Other Comments / Reviews
SaaS Metrics