Comment on: w8a16 (int8 GPTQ) target: Marlin repack CUDA IMA (wedges card until reboot); every fallback route fails — Conch 23 tok/s, fp16 blocked by bf16-only spec-decode kernels
Repo: syv-ai/qwen38-27b-rtx3090 by mhenrichsen
**Correcting myself on row 5.** I said I had taken your rejection-sampler assert at face value as a second bf16 assumption and had not read that kernel. I have now, and **it is not one.**
`rejection_sampler_utils.py` contains **zero** `bfloat16`/`float16` literals. Every load promotes to `tl.float32`, the index arithmetic is `tl.int64`, and the intermediates are allocated off `target_logits.new_empty(..., dtype=torch.float32)`. The sampler is dtype-agnostic — it cannot care what your activations are.
So the assert at `rejection_sampler_utils.py:931` has some other cause. A device-side assert is the signature of an out-of-bounds index rather than a dtype mismatch (the bounds-checked loads in there are masked on `vocab_size`, and the per-request loops run to `num_draft_tokens = end_idx - start_idx - 1`), and your row 5 was Exllama on a w8a16 target whose GEMM path had already failed three other ways in rows 1-4. **Garbage logits or a bad token id out of a broken linear path would produ...
GitHub Issue
SaaS Metrics