SaaS Metrics
Insight for: w8a16 (int8 GPTQ) target: Marlin repack CUDA IMA (wedges card until reboot); every fallback route fails — Conch 23 tok/s, fp16 blocked by bf16-only spec-decode kernels
Compatibility and stability of w8a16 (GPTQ int8) quantized models with various linear kernels (Marlin, Exllama, Conch) and speculative decoding (DFlash2) on NVIDIA CMP 170HX, specifically addressing CUDA illegal memory access (IMA) and dtype mismatches.
This report exposes severe stability and compatibility issues when deploying w8a16 GPTQ-quantized models on NVIDIA CMP 170HX hardware. Critical failures include CUDA illegal memory access (IMA) that wedges the GPU, and dtype mismatches blocking fp16 execution due to hardcoded bf16-only kernels. While a workaround for the IMA (sm80 VMM corruption) was found, the pervasive nature of these errors across different linear kernels (Marlin, Exllama) and the drastic performance degradation with Conch (23 tok/s vs 133.7 tok/s) highlight significant technical debt. This directly impacts the viability of deploying highly optimized, quantized models on diverse hardware, increasing development overhead and limiting market reach for performance-critical B2B applications.