SaaS Metrics
GitHub Issue
Known issues and progress: KVarN / DFlash2 / CTX=huge (tracking)
Running list of what is broken, what is fixed, and what is only mitigated, so nobody has to reconstruct it from the [#13](https://github.com/syv-ai/qwen38-27b-rtx3090/pull/13) thread. Most of this came out of that PR and the reproductions @mjungnickel18, @Dmtrii-tesla and @alexander99lab ran on hardware I do not have.
I will keep this updated as things move.
## Fixed
| What | Commit | Notes |
|---|---|---|
| **Bug B** — output corrupted at one prompt length in every 128 | `a75ee4b` | `get_uniform_token_count` has no prefill discriminator, so a final prefill chunk of exactly `decode_query_len` tokens was dispatched as a uniform *decode* batch and replayed the captured spec-verify graph. Needs a prefix-cache hit, because only a hit makes the scheduler cut the tail at `prompt_len % 128`. Rule: breaks iff templated prompt length ≡ `L` (mod 128), `L = k+1`. |
| **`SPEC=mtp CTX=huge` shipped with Bug B** | `a75ee4b`, `674e944` | We had scoped the workaround to `dflash2` on the theory that MTP's short verify step captures correctly. It does not. |
| **#18** — `DFLASH_TOKENS=3` + `CTX=huge` + `PREFIX_CACHE=1` killed the engine on the first request | `674e944` | The sliding-window guard only handled the SW block being *smaller* than the hash unit. At k=3 the page arithmetic promotes it *larger* (2048 vs a 128 unit), which slipped past and tripped an upstream assert. |
| **`--prefix-match-unit 128` is not a TP=2 workaround** | merged in `f91c447` | @Dmtrii-tesla. Needed on a single ...
View Raw Thread
Developer & User Discourse
mhenrichsen • Aug 22, 2026
**Update — two of the three open items are closed, both verified on hardware.** Pushed in `82bd62d`.
## `DFLASH_TOKENS=15` does not boot at 240k → **fixed**
My first fix was aimed at the wrong thing. I had found that `CTX=huge` was the only branch never raising `VLLM_V2_CUDAGRAPH_MEM_MIB` above 7 drafts — true, and worth fixing — and assumed it explained the boot failure. It does not: `KV_MEM` is pinned, so accounting graph memory differently cannot grow the pool. vLLM says it outright:
```
To serve at least one request with the model's max seq len (245760), 5.16 GiB KV cache
is needed, which is larger than the available KV cache memory (4.9 GiB)
```
The real fix is the trade `CTX=fast` already makes above 7 drafts — cap `max_model_len`. 229,376 was *still* 1% short (4.95 vs 4.90 GiB); **221,184** boots, with a 223,974-token pool, 794/794 verbatim at two residues cold and warm, 14.71 tok/step.
@alexander99lab — this should mean `DFLASH_TOKENS=15` now starts on your 24 GB card. Wor...
## `DFLASH_TOKENS=15` does not boot at 240k → **fixed**
My first fix was aimed at the wrong thing. I had found that `CTX=huge` was the only branch never raising `VLLM_V2_CUDAGRAPH_MEM_MIB` above 7 drafts — true, and worth fixing — and assumed it explained the boot failure. It does not: `KV_MEM` is pinned, so accounting graph memory differently cannot grow the pool. vLLM says it outright:
```
To serve at least one request with the model's max seq len (245760), 5.16 GiB KV cache
is needed, which is larger than the available KV cache memory (4.9 GiB)
```
The real fix is the trade `CTX=fast` already makes above 7 drafts — cap `max_model_len`. 229,376 was *still* 1% short (4.95 vs 4.90 GiB); **221,184** boots, with a 223,974-token pool, 794/794 verbatim at two residues cold and warm, 14.71 tok/step.
@alexander99lab — this should mean `DFLASH_TOKENS=15` now starts on your 24 GB card. Wor...
mjungnickel18 • Aug 22, 2026
Our comments crossed — mine went up six minutes before yours, so this is about one line in your update rather than the conclusion.
You landed on `mtp` keeping PIECEWISE, which is where I would land too. But the stated reason is *"both speculators are now correct under FULL — five residues each"*, and correctness is the part I measure differently. From the table above, six fresh servers, one warm-up request each and one measured request:
| `SPEC=mtp CTX=huge`, `3059e35` | residue 3 | **residue 4** | residue 5 |
|---|---|---|---|
| `FULL_AND_PIECEWISE` | 1150/1150 | **1/2, `stop`, 1.00 tok/step** | 1150/1150 |
| `PIECEWISE` | 1150/1150 | 1150/1150 | 1150/1150 |
**Which five residues?** At one broken length in 128, five samples miss it about 96% of the time, and `L = k+1 = 4` is the one that has to be in the set. Sweeping the configuration across all 128 residues gives 127 clean and exactly one broken, and it is 4.
If your five did include 4 and came back clean, then we disagree about...
You landed on `mtp` keeping PIECEWISE, which is where I would land too. But the stated reason is *"both speculators are now correct under FULL — five residues each"*, and correctness is the part I measure differently. From the table above, six fresh servers, one warm-up request each and one measured request:
| `SPEC=mtp CTX=huge`, `3059e35` | residue 3 | **residue 4** | residue 5 |
|---|---|---|---|
| `FULL_AND_PIECEWISE` | 1150/1150 | **1/2, `stop`, 1.00 tok/step** | 1150/1150 |
| `PIECEWISE` | 1150/1150 | 1150/1150 | 1150/1150 |
**Which five residues?** At one broken length in 128, five samples miss it about 96% of the time, and `L = k+1 = 4` is the one that has to be in the set. Sweeping the configuration across all 128 residues gives 127 clean and exactly one broken, and it is 4.
If your five did include 4 and came back clean, then we disagree about...
mjungnickel18 • Aug 22, 2026
Two data points from sweeping our own deployment rather than a test tree, plus one loose end to close.
## `--prefix-match-unit` is the discriminator, not the block size
Our two production services differ in exactly one relevant way, so together with @alexander99lab's `CTX=fast` negative they make a three-row ladder. All on `30d8116` (pre-`a75ee4b`), `FULL_AND_PIECEWISE`, prefix caching on, 128/128 residues swept:
| geometry | Bug B | source |
|---|---|---|
| block 16, no `--prefix-match-unit` | clean | @alexander99lab, `CTX=fast` |
| **block 128, no `--prefix-match-unit`** | **clean** | this box, `SPEC=mtp` service |
| block 128, **with** `--prefix-match-unit 128` | **broken at residue 8** | this box, `SPEC=dflash2` service |
So it is the match unit that arms it, not the block size — which is what your mechanism predicts, since only the unit makes the scheduler cut the tail at `len % 128`. The middle row is the one nobody had.
## Bug B was live in a real deployment, silently
Wort...
## `--prefix-match-unit` is the discriminator, not the block size
Our two production services differ in exactly one relevant way, so together with @alexander99lab's `CTX=fast` negative they make a three-row ladder. All on `30d8116` (pre-`a75ee4b`), `FULL_AND_PIECEWISE`, prefix caching on, 128/128 residues swept:
| geometry | Bug B | source |
|---|---|---|
| block 16, no `--prefix-match-unit` | clean | @alexander99lab, `CTX=fast` |
| **block 128, no `--prefix-match-unit`** | **clean** | this box, `SPEC=mtp` service |
| block 128, **with** `--prefix-match-unit 128` | **broken at residue 8** | this box, `SPEC=dflash2` service |
So it is the match unit that arms it, not the block size — which is what your mechanism predicts, since only the unit makes the scheduler cut the tail at `len % 128`. The middle row is the one nobody had.
## Bug B was live in a real deployment, silently
Wort...
mhenrichsen • Aug 22, 2026
@mjungnickel18 **you are right and I was wrong.** Reproduced on current main, fresh server, the three residues you named:
| `SPEC=mtp CTX=huge` FULL | cold | self-hit |
|---|---|---|
| residue 3 | 794/794 | 794/794 |
| **residue 4** | **empty** | **1 char, then empty** |
| residue 5 | 794/794 | 794/794 |
`sha da39a3ee5e6b` is sha1 of the empty string. `finish_reason=stop`, `tok/step` 1.00 or undefined.
My five residues were 8, 12, 16, 32 and 124. **Four was not among them**, and five samples miss a single broken residue 82% of the time — so "5/5 clean" was never evidence for the claim I hung on it, whatever the answer turned out to be. Your 128/128 sweep is the right standard and I have adopted it; a full sweep of `dflash2 CTX=huge` under FULL is running now, because I gave *that* one FULL on the same inadequate sample and owe it the same test.
**Your governance point is the part I have acted on.** The launcher comment now records the mtp line as a correctness constraint with the r...
| `SPEC=mtp CTX=huge` FULL | cold | self-hit |
|---|---|---|
| residue 3 | 794/794 | 794/794 |
| **residue 4** | **empty** | **1 char, then empty** |
| residue 5 | 794/794 | 794/794 |
`sha da39a3ee5e6b` is sha1 of the empty string. `finish_reason=stop`, `tok/step` 1.00 or undefined.
My five residues were 8, 12, 16, 32 and 124. **Four was not among them**, and five samples miss a single broken residue 82% of the time — so "5/5 clean" was never evidence for the claim I hung on it, whatever the answer turned out to be. Your 128/128 sweep is the right standard and I have adopted it; a full sweep of `dflash2 CTX=huge` under FULL is running now, because I gave *that* one FULL on the same inadequate sample and owe it the same test.
**Your governance point is the part I have acted on.** The launcher comment now records the mtp line as a correctness constraint with the r...
## 1. The MTP break reproduces with a fresh server per data point
You wrote that reproductions with a fresh server per point are worth several times a warm sweep, and my previous evidence was one server for five lengths. Redone on `3059e35`, six servers, each booted fresh, one warm-up request to arm the cache (ctx 25024) and exactly one measured request, then killed:
| `SPEC=mtp CTX=huge PREFIX_CACHE=1` | residue 3 | **residue 4** | residue 5 |
|---|---|---|---|
| `FULL_AND_PIECEWISE` | 1150/1150 | **1/2, `finish_reason=stop`, 1.00 tok/step** | 1150/1150 |
| `PIECEWISE` | 1150/1150 | 1150/1150, 3.76 tok/step | 1150/1150 |
Residue 4 is `L = k+1` for MTP's verify block at `CTX=huge` (`DRAFT_TOKENS=3`). One cell of six, and it is the one the rule points a...