ROIpad ← Back to Search

Insight for: Experiment: Fused Q·Centroid compressed attention for turbo3 decode

`turbo3` decode performance for LLM inference on Apple Silicon (M1, M2 Pro, M5 Max), specifically addressing the 'decode cliff' at increasing context depths.

Analyzed: Apr 1, 2026

This extensive analysis identifies a critical performance bottleneck for `turbo3` decode on Apple Silicon: a 'decode cliff' at increasing context depths, particularly severe on M1/M2, initially attributed to centroid LUT constant memory accesses. Profiling reveals the constant memory LUT is indeed a significant factor, performing 2x worse on M2 than M5. However, the core issue is not the constant cache itself, but the *cost* of LUT lookups and related operations. Solutions like 'Batched Extract' and a '4-Entry Magnitude LUT + Branchless Sign' significantly improve M2 Pro decode performance, demonstrating that targeted micro-optimizations for specific hardware architectures are crucial. For B2B SaaS, consistent performance across diverse hardware, especially for long-context LLM inference, is a key differentiator. This deep dive into hardware-specific bottlenecks underscores the necessity of low-level optimization to unlock full performance potential and maintain competitive advantage.

turbo3 decode data-dependent constant memory accesses centroid LUT lookup L2 cache pressure decode ratio curve q8_0 context depths half cn[8] registers Metal Threadgroup centroid cache Per-block norm*centroid table cn_norm PPL Bit-Arithmetic Dequant M2 decode cliff constant LUT ALU cost Batched Extract bit field pre-extraction TURBO_PROFILE_MODE Profiling isolation No-op turbo3 compressed cache bandwidth 4-Entry Magnitude LUT Branchless Sign XOR trick Qwen2.5-7B-Instruct-Q4_K_M

GitHub Issue

Parent Entity

Experiment: Fused Q·Centroid compressed attention for turbo3 decode

State: Open