SaaS Metrics
Gemini Executive Synthesis
Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations.
Technical Positioning
Ensuring correctness and stability for Qwen3.8-27B inference with advanced speculative decoding (MTP, DFlash2) and extended context lengths (CTX=huge) on RTX 3090, by systematically identifying and fixing critical bugs related to prefill handling, sliding window guards, and KV cache allocation.
SaaS Insight & Market Implications
This tracking issue details critical bug fixes for Qwen3.8-27B's advanced inference configurations, including output corruption, engine crashes, and KV cache memory constraints. The resolution of issues like 'Bug B' (output corruption tied to prompt length modulo 128) and engine kills due to sliding-window guard logic directly enhances model reliability. The fix for `DFLASH_TOKENS=15` boot failures by adjusting `max_model_len` addresses practical deployment limits on 24GB cards. These fixes are crucial for production environments, demonstrating a commitment to stability and performance under complex speculative decoding and large context scenarios. The ongoing validation of correctness, particularly for MTP under `FULL_AND_PIECEWISE` mode, highlights the continuous effort required to maintain high-fidelity LLM operations.
Proprietary Technical Taxonomy
Raw Developer Origin & Technical Request
GitHub Issue
Aug 22, 2026
Repo: syv-ai/qwen38-27b-rtx3090
Known issues and progress: KVarN / DFlash2 / CTX=huge (tracking)
Running list of what is broken, what is fixed, and what is only mitigated, so nobody has to reconstruct it from the [#13](github.com/syv-ai/qwen38-27b... thread. Most of this came out of that PR and the reproductions @mjungnickel18, @Dmtrii-tesla and @alexander99lab ran on hardware I do not have.
I will keep this updated as things move.
## Fixed
| What | Commit | Notes |
|---|---|---|
| **Bug B** — output corrupted at one prompt length in every 128 | `a75ee4b` | `get_uniform_token_count` has no prefill discriminator, so a final prefill chunk of exactly `decode_query_len` tokens was dispatched as a uniform *decode* batch and replayed the captured spec-verify graph. Needs a prefix-cache hit, because only a hit makes the scheduler cut the tail at `prompt_len % 128`. Rule: breaks iff templated prompt length ≡ `L` (mod 128), `L = k+1`. |
| **`SPEC=mtp CTX=huge` shipped with Bug B** | `a75ee4b`, `674e944` | We had scoped the workaround to `dflash2` on the theory that MTP's short verify step captures correctly. It does not. |
| **#18** — `DFLASH_TOKENS=3` + `CTX=huge` + `PREFIX_CACHE=1` killed the engine on the first request | `674e944` | The sliding-window guard only handled the SW block being *smaller* than the hash unit. At k=3 the page arithmetic promotes it *larger* (2048 vs a 128 unit), which slipped past and tripped an upstream assert. |
| **`--prefix-match-unit 128` is not a TP=2 workaround** | merged in `f91c447` | @Dmtrii-tesla. Needed on a single ...
Developer Debate & Comments
Two follow-ups to [my comment on ](https://github.com/syv-ai/qwen38-27b-rtx3090/pull/13#issuecomment-5379687913), moved here since that PR is closed: the MTP result re-run to your fresh-server standard, and a retraction of my own. ## 1. The MTP break reproduces with a fresh server per data point You wrote that reproductions with a fresh server per point are worth several times a warm sweep, and my previous evidence was one server for five lengths. Redone on `3059e35`, six servers, each booted fresh, one warm-up request to arm the cache (ctx 25024) and exactly one measured request, then killed: | `SPEC=mtp CTX=huge PREFIX_CACHE=1` | residue 3 | **residue 4** | residue 5 | |---|---|---|---| | `FULL_AND_PIECEWISE` | 1150/1150 | **1/2, `finish_reason=stop`, 1.00 tok/step** | 1150/1150 | | `PIECEWISE` | 1150/1150 | 1150/1150, 3.76 tok/step | 1150/1150 | Residue 4 is `L = k+1` for MTP's verify block at `CTX=huge` (`DRAFT_TOKENS=3`). One cell of six, and it is the one the rule points a...
**Update — two of the three open items are closed, both verified on hardware.** Pushed in `82bd62d`. ## `DFLASH_TOKENS=15` does not boot at 240k → **fixed** My first fix was aimed at the wrong thing. I had found that `CTX=huge` was the only branch never raising `VLLM_V2_CUDAGRAPH_MEM_MIB` above 7 drafts — true, and worth fixing — and assumed it explained the boot failure. It does not: `KV_MEM` is pinned, so accounting graph memory differently cannot grow the pool. vLLM says it outright: ``` To serve at least one request with the model's max seq len (245760), 5.16 GiB KV cache is needed, which is larger than the available KV cache memory (4.9 GiB) ``` The real fix is the trade `CTX=fast` already makes above 7 drafts — cap `max_model_len`. 229,376 was *still* 1% short (4.95 vs 4.90 GiB); **221,184** boots, with a 223,974-token pool, 794/794 verbatim at two residues cold and warm, 14.71 tok/step. @alexander99lab — this should mean `DFLASH_TOKENS=15` now starts on your 24 GB card. Wor...
Our comments crossed — mine went up six minutes before yours, so this is about one line in your update rather than the conclusion. You landed on `mtp` keeping PIECEWISE, which is where I would land too. But the stated reason is *"both speculators are now correct under FULL — five residues each"*, and correctness is the part I measure differently. From the table above, six fresh servers, one warm-up request each and one measured request: | `SPEC=mtp CTX=huge`, `3059e35` | residue 3 | **residue 4** | residue 5 | |---|---|---|---| | `FULL_AND_PIECEWISE` | 1150/1150 | **1/2, `stop`, 1.00 tok/step** | 1150/1150 | | `PIECEWISE` | 1150/1150 | 1150/1150 | 1150/1150 | **Which five residues?** At one broken length in 128, five samples miss it about 96% of the time, and `L = k+1 = 4` is the one that has to be in the set. Sweeping the configuration across all 128 residues gives 127 clean and exactly one broken, and it is 4. If your five did include 4 and came back clean, then we disagree about...
Two data points from sweeping our own deployment rather than a test tree, plus one loose end to close. ## `--prefix-match-unit` is the discriminator, not the block size Our two production services differ in exactly one relevant way, so together with @alexander99lab's `CTX=fast` negative they make a three-row ladder. All on `30d8116` (pre-`a75ee4b`), `FULL_AND_PIECEWISE`, prefix caching on, 128/128 residues swept: | geometry | Bug B | source | |---|---|---| | block 16, no `--prefix-match-unit` | clean | @alexander99lab, `CTX=fast` | | **block 128, no `--prefix-match-unit`** | **clean** | this box, `SPEC=mtp` service | | block 128, **with** `--prefix-match-unit 128` | **broken at residue 8** | this box, `SPEC=dflash2` service | So it is the match unit that arms it, not the block size — which is what your mechanism predicts, since only the unit makes the scheduler cut the tail at `len % 128`. The middle row is the one nobody had. ## Bug B was live in a real deployment, silently Wort...
@mjungnickel18 **you are right and I was wrong.** Reproduced on current main, fresh server, the three residues you named: | `SPEC=mtp CTX=huge` FULL | cold | self-hit | |---|---|---| | residue 3 | 794/794 | 794/794 | | **residue 4** | **empty** | **1 char, then empty** | | residue 5 | 794/794 | 794/794 | `sha da39a3ee5e6b` is sha1 of the empty string. `finish_reason=stop`, `tok/step` 1.00 or undefined. My five residues were 8, 12, 16, 32 and 124. **Four was not among them**, and five samples miss a single broken residue 82% of the time — so "5/5 clean" was never evidence for the claim I hung on it, whatever the answer turned out to be. Your 128/128 sweep is the right standard and I have adopted it; a full sweep of `dflash2 CTX=huge` under FULL is running now, because I gave *that* one FULL on the same inadequate sample and owe it the same test. **Your governance point is the part I have acted on.** The launcher comment now records the mtp line as a correctness constraint with the r...
Adjacent Repository Pain Points
Other highly discussed features and pain points extracted from syv-ai/qwen38-27b-rtx3090.
Extracted Positioning
Compatibility and stability of w8a16 (GPTQ int8) quantized models with various linear kernels (Marlin, Exllama, Conch) and speculative decoding (DFlash2) on NVIDIA CMP 170HX, specifically addressing CUDA illegal memory access (IMA) and dtype mismatches.
Achieving robust, high-performance inference for GPTQ int8 quantized Qwen3.8-27B models on specialized hardware (CMP 170HX) by ensuring compatibility across different linear kernels and resolving critical dtype-related failures and memory corruption issues.
Top Replies
This is an unusually good report — five attempts with the blame isolated by controls, and a working-vs-broken vLLM version to bisect against. Taking your two asks in reverse order, because I can ac...
**Correcting myself on row 5.** I said I had taken your rejection-sampler assert at face value as a second bf16 assumption and had not read that kernel. I have now, and **it is not one.** `rejectio...
**Root cause found and worked around on our 170HX — the IMA is sm80 VMM corruption under allocation churn, not a kernel pointer bug.** Full story below; patch linked at the end. ## What actually fa...
Extracted Positioning
Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains.
Optimizing multi-GPU inference performance for Qwen3.8-27B on cost-effective hardware (dual RTX 3090s) to achieve significant throughput improvements (+16-35%) across various concurrency levels and speculative decoding methods (MTP, DFlash2), while resolving memory allocation inefficiencies.
Top Replies
This is really useful information. I'm also running a very similar setup and had planned to provide some numbers this weekend but the information you've provided is so much more in depth. Also, you...
I've changed my configuration to roughly match your suggestions and (warm) benched at 250W and 275W. My numbers are significantly better than they were though a little lower than yours at C1. I sus...
This is the best-run field report this repo has received — a genuine controlled A/B, spread stated, negative results kept, and the two findings that touch deliberate calibrations flagged instead of...
Extracted Positioning
Functionality and performance of OffloadingConnector (KV eviction to RAM/disk) for extending KV cache capacity on the Qwen3.8-27B stack.
Enabling deployment of Qwen3.8-27B with extended context lengths beyond GPU VRAM limits by leveraging system RAM/disk for KV cache, even in virtualized environments (Proxmox VM with GPU passthrough).
Top Replies
Two follow-ups from running this in anger. **Edge 6 escalated from ops-hygiene to a production outage.** The engine died once (host OOM, most likely — nothing in the engine log), and then `restart:...
Answers to all three open questions, one shipped fix for the outage class, and one debt I am keeping. Your follow-up comment upgraded this report from useful to load-bearing — a 70-restart ghost lo...
The traced cell I owed you is done — your "store path works, load path never hits" is **reproduced on this card, mechanism found, and it is neither unliftable-by-design nor a hash miss**. It's an a...
Extracted Positioning
Stability of vLLM EngineCore with flashinfer attention backend and MTP speculative decoding, specifically concerning CUDA illegal memory access errors during long context processing.
Ensuring robust and error-free inference for Qwen3.8-27B (W4A16 AutoRound) on RTX 3090, particularly with advanced features like flashinfer and MTP, even with long contexts (~28k tokens).
Top Replies
Good report — the dump_input capture and the dmesg line are exactly what makes this analyzable from a distance. One red herring to clear first, then where the real suspicion points, then what I wou...
## Follow-up: second occurrence on 2026-08-27, same fault address The same crash recurred today (2026-08-27 14:03:48 local, after ~40h uptime, docker restart policy auto-recovered). **New data poin...
## Test result: option 1 (`--attention-backend FLASH_ATTN`) is hard-invalid on the fp8-KV tier As suggested, I added `--attention-backend FLASH_ATTN` to the `CTX=long` launch (everything else uncha...
Extracted Positioning
System RAM requirements for loading and running Qwen3.8-27B with vLLM, specifically addressing limitations with 16GB main system RAM.
Enabling deployment of Qwen3.8-27B on systems with constrained main memory, potentially through optimized loading strategies or reduced memory footprint.
Top Replies
Honest framing first: 16 GB of host RAM (10 GiB actually free) is below anything this stack has been run on, so what follows is diagnosis plus the two levers that exist — not a tested recipe. It ma...
> system ram is 16GB, using iGPU Well, your iGPU is using system RAM; what OS & have you tried to reduce background processes? Close as many web browser tabs as you can, & you might have to use thi...
Frequently Asked Questions
Market intelligence mapped to Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations..
What problem does Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations. solve?
Based on our AI analysis of the original developer request, its primary technical positioning is: Ensuring correctness and stability for Qwen3.8-27B inference with advanced speculative decoding (MTP, DFlash2) and extended context lengths (CTX=huge) on RTX 3090, by systematically identifying and fixing critical bugs related to prefill handling, sliding window guards, and KV cache allocation.
Are engineers actively discussing Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations.?
Yes, we have tracked 19 direct responses and active debates regarding this specific topic originating from GitHub Issue.
What architecture is tied to Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations.?
Our proprietary extraction maps Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations. to adjacent architectural concepts including KVarN / DFlash2 / CTX=huge, output corrupted at one prompt length in every 128, prefill discriminator, spec-verify graph.
Engagement Signals
Cross-Market Term Frequency
Quantifies the cross-market adoption of foundational terms like PIECEWISE and DRAFT_TOKENS=3 by tracking occurrence frequency across active SaaS architectures and enterprise developer debates.