ROIpad ← Back to Search
github.com › issue comment

Comment on: w8a16 (int8 GPTQ) target: Marlin repack CUDA IMA (wedges card until reboot); every fallback route fails — Conch 23 tok/s, fp16 blocked by bf16-only spec-decode kernels

Repo: syv-ai/qwen38-27b-rtx3090 by mhenrichsen
Posted: Aug 26, 2026
@ahnguyen17 The candidate patch I owed you is on main: [`patches/marlin-repack-staged-sm80.patch`](https://github.com/syv-ai/qwen38-27b-rtx3090/blob/main/patches/marlin-repack-staged-sm80.patch) (commit `3cc6afc`). ## Design — "keep the intermediates small", taken literally Rather than chunking the repack math (the Marlin tile layout makes column-slab independence a property I'd have to prove about the kernel, not assume), it removes the allocation churn itself: the `.contiguous()` copy and the pad — the two per-layer transients around `gptq_marlin_repack` — now land in **one persistent, grow-only staging buffer per device**, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic collapses from two transients + one output to the output alone; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, vs hundreds of alloc/free pairs). The repack kernel itself is untouched and receives identical b...