Comment on: Engineering findings: K/V norm disparity + MSE > Prod + outlier mixed precision
Repo: TheTom/turboquant_plus by TheTom
update on this: we ran a full turbo4 investigation this week and your MSE > Prod finding is now independently confirmed on three setups:
1. our Metal (M5 Max): QJL ablation on turbo4 shows removing QJL improves PPL from 6.1894 to 6.1756. QJL actively hurts.
2. buun's CUDA (RTX 3090): turbo4 degrades from -0.28% at 2K to +3.69% at 64K. QJL noise accumulates with context.
3. your GPT-2 data: Prod at +300% PPL vs MSE at +7.6%.
we've dropped QJL from turbo4 entirely and fixed the dequant path (byte-aligned packing, direct extraction). turbo4 now matches turbo3 speed and quality. the QJL bit was pure waste.
next step for us is looking at asymmetric K/V using your norm disparity data as a starting point.
GitHub Issue
Parent Entity
State: Open • Comments: 2
Other Comments / Reviews
SaaS Metrics