syv-ai/qwen38-27b-rtx3090

## Test result: option 1 (`--attention-backend FLASH_ATTN`) is hard-invalid on the fp8-KV tier

As suggested, I added `--attention-backend FLASH_ATTN` to the `CTX=long` launch (everything else unchanged: fp8 KV, MTP k=3, prefix caching, chunked prefill, max_seq_len=150k, vLLM 0.27.1, single RTX 3090) and force-recreated the container.

**Result: the engine rejects the combination at startup — a validation error, not a runtime crash.** The container crash-looped four times over ~5 minutes (each attempt lived ~100s until the error fired), then I reverted. Startup log:

```
ValueError: Selected backend AttentionBackendEnum.FLASH_ATTN is not valid for this configuration.
Reason: ['kv_cache_dtype not supported', 'FP8 KV cache requires FA3 on SM90 or FA4 on SM100']
[rank0]:[W827 17:24:27.168095284 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources.
(APIServer pid=1) INFO 08-27 17:24:28 [utils.py:612] [shutdown] Pr...

## Follow-up: second occurrence on 2026-08-27, same fault address

The same crash recurred today (2026-08-27 14:03:48 local, after ~40h uptime, docker restart policy auto-recovered).

**New data point that changes the hypothesis:** the dmesg MMU fault is at the **exact same virtual address** as the first crash on 2026-08-24:

```
NVRM: Xid (PCI:0000:01:00): 31, pid=, name=VLLM::EngineCor, channel 0x00000008, intr 00000000.
MMU Fault: ENGINE GRAPHICS GPC1 GPCCLIENT_T1_0 faulted @ 0xd_2892f000.
Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
```

Both crashes:
- identical fault address `0xd_2892f000`, identical GPC1 GPCCLIENT_T1_0, identical `FAULT_PDE ACCESS_TYPE_VIRT_WRITE` (page-directory-entry level write fault)
- identical crash point: `_build_attention_metadata` -> flashinfer `build()` -> `seq_lens.to("cpu")` raising `torch.AcceleratorError: CUDA error: an illegal memory access was encountered`

This rules out a random/one-off OOB write: a **deterministic, repeating f...

No problem. I kept everything else the same and only made the .env changes you specified. I know you said C1 was all that is needed but I just ran the same bench script as it's nice for me to be able to compare in full.

Still running the same commit from the previous run: https://github.com/syv-ai/qwen38-27b-rtx3090/commit/60daef8255b6757d9791955a44bce27df1658ea6

**LOOKUP=0 DFLASH_TOKENS=15 (otherwise exact same as previous .env)**

| Cohort | Temp | Decode (tok/s) | e2e (tok/s) | Tokens/Step | Mean TTFT (ms) |
|---|---|---|---|---|---|
| C1 | default | 134.2 | 130.56 | 3.45 | 135.77 |
| C2 | default | 178.7 | 161.62 | 3.19 | 188.14 |
| C4 | default | 240.0 | 203.10 | 3.43 | 305.49 |
| C8 | default | 257.2 | 205.92 | 3.13 | 721.15 |
| C1 | greedy (T=0) | 130.5 | 126.02 | 3.30 | 140.17 |
| C2 | greedy (T=0) | 191.2 | 166.35 | 3.32 | 185.53 |
| C4 | greedy (T=0) | 243.8 | 208.12 | 3.45 | 288.03 |
| C8 | greedy (T=0) | 289.2 | 231.50 | 3.49 | 730.25 |

For this one I'm posting the ful...

This is the best-run field report this repo has received — a genuine controlled A/B, spread stated, negative results kept, and the two findings that touch deliberate calibrations flagged instead of papered over. Both of those findings are correct, and both are now fixed on main. Point by point, then what I owe you, then @dpcartwright's T=15 result, which I think is the most interesting datapoint in the thread.

## 4.3 — the KV_MEM pin: you are right, and the launcher now skips it at TP>1

The pin is a single-card constant (the 24 GiB transient margin is sharp — 396 MiB dead / 436 MiB fine, gotcha 39), and vLLM applies `--kv-cache-memory` per worker, so at TP=2 it strands ~8 GiB per card exactly as you measured. As of `60921ed` the launcher detects `--tensor-parallel-size`/`-tp` in `EXTRA_ARGS` and, unless you exported a `KV_MEM` yourself, sizes the pool from `GPU_UTIL` with a note — your suggested fix, taken as suggested. Your unpin A/B (302,223 tokens for deltas inside spread, boots c...

The traced cell I owed you is done — your "store path works, load path never hits" is **reproduced on this card, mechanism found, and it is neither unliftable-by-design nor a hash miss**. It's an allocator geometry problem, and your own bf16-vs-KVarN split falls out of it exactly. Fixes and a boot-time warning are on main; the full story so you can check it against your VM.

## The reproduction

KVarN `CTX=huge` + your exact `TieringOffloadingSpec` connector config (4 GiB CPU tier), instrumented at `CPUOffloadingManager.{lookup,prepare_store,complete_store,touch}` and the connector's key builder. Protocol: document A (~23.5k tokens) → B → C (fill the GPU pool, evict A) → A again. Result: `A first=22.2s, resend=22.3s` — full recompute, `CPU_to_GPU` zero. Your symptom, on bare metal, no VM in the picture.

## The mechanism, from the trace

The store side is *perfect*: A's chunks are stored and `complete_store(success=True)` lands for every key (in one capture the very key the resend late...

@ahnguyen17 The candidate patch I owed you is on main: [`patches/marlin-repack-staged-sm80.patch`](https://github.com/syv-ai/qwen38-27b-rtx3090/blob/main/patches/marlin-repack-staged-sm80.patch) (commit `3cc6afc`).

## Design — "keep the intermediates small", taken literally

Rather than chunking the repack math (the Marlin tile layout makes column-slab independence a property I'd have to prove about the kernel, not assume), it removes the allocation churn itself: the `.contiguous()` copy and the pad — the two per-layer transients around `gptq_marlin_repack` — now land in **one persistent, grow-only staging buffer per device**, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic collapses from two transients + one output to the output alone; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, vs hundreds of alloc/free pairs). The repack kernel itself is untouched and receives identical b...

I've changed my configuration to roughly match your suggestions and (warm) benched at 250W and 275W. My numbers are significantly better than they were though a little lower than yours at C1. I suspect there may be some thermal throttling going on as I've not had this setup for long and probably need to tweak things.

I'll also add that I'm fairly new to running local models and vLLM in general so please feel free to point out any rookie errors.

Setup:

- Ubuntu 26.04
- Docker deployment
- 10920x, 128GB DDR4, Dual 3090 (EVGA Super FTW3), 16x PCIE3 for both cards, no nvlink, p2p driver patch
- Commit: https://github.com/syv-ai/qwen38-27b-rtx3090/commit/60daef8255b6757d9791955a44bce27df1658ea6

My new .env:
```env
VLLM_API_KEY=REMOVED
CUDA_VISIBLE_DEVICES=0,1
GPU_UTIL=0.93
CTX=fast
MAX_LEN=294912
MAX_SEQS=8
SPEC=dflash2
PREFIX_CACHE=1
VISION=1
KV_MEM=
DFLASH_TOKENS=7

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

EXTRA_ARGS='--tensor-parallel-size 2 --disable-custom-all-reduce --hf-overrides {"t...

This is really useful information. I'm also running a very similar setup and had planned to provide some numbers this weekend but the information you've provided is so much more in depth. Also, your numbers are far better than mine so I will spend some time comparing configurations and report back if I can tweak things and achieve the same results.

To give a very rough idea of what I'm seeing: 10920x, 128GB quad channel DDR4, 2 x 3090 over dual 16x PCIE3 with p2p (but no nvlink). Caps out at around 122 tok/sec with ~300k context running CTX=fast.

> system ram is 16GB, using iGPU

Well, your iGPU is using system RAM; what OS & have you tried to reduce background processes?
Close as many web browser tabs as you can, & you might have to use this computer as an AI-server-only computer, & do you work on a second (cheap) machine....

Honest framing first: 16 GB of host RAM (10 GiB actually free) is below anything this stack has been run on, so what follows is diagnosis plus the two levers that exist — not a tested recipe. It may well be workable, because weight loading is the only phase that touches checkpoint-sized data on the host; but the log you pasted cuts off exactly where the answer is.

## What I need to see

The failure at shard 5/8 tells me *when*, not *why*. Three things decide it:

1. **The full traceback after `EngineCore failed to start`** — the exception type is the whole diagnosis (`MemoryError`, `OSError: [Errno 12] Cannot allocate memory` from an mmap, or nothing at all because the process was killed).
2. **`dmesg -T | grep -iE "oom|killed process"`** right after a failed attempt — if the OOM killer took it, this says so and names the RSS at death.
3. **Where the checkpoint lives and what wraps vLLM**: is `models/` on a real disk (your log says EXT4, good — but confirm it is not a tmpfs/zram-backe...

Good report — the dump_input capture and the dmesg line are exactly what makes this analyzable from a distance. One red herring to clear first, then where the real suspicion points, then what I would change and what I would like from you.

## The `[-1, -1, -1]` in your scheduler dump is not corruption

Those are deliberate padding, from source: when drafts are trimmed (grammar validation, chunked-prefill limits, or async-scheduling timing), the scheduler pads `spec_token_ids` back to the scheduled count with `-1` and records the count in `num_invalid_spec_tokens` — the runner masks them, the stats subtract them. A step scheduled as `[-1, -1, -1]` is a legal "no usable drafts this step" state, not evidence of the fault. So the dump does not implicate the MTP proposer, and I would drop that thread.

## Where your stack differs from the one this repo tests

Your config is close to `CTX=long` geometry but differs in one load-bearing place: **`--attention-backend flashinfer`**. The launcher...

Answers to all three open questions, one shipped fix for the outage class, and one debt I am keeping. Your follow-up comment upgraded this report from useful to load-bearing — a 70-restart ghost loop found in production is exactly the kind of evidence a boot-time fix needs to justify itself.

## Edge 6: shipped in `60daef8`

Both launchers now unlink stale `/dev/shm/vllm_offload_*.mmap` regions **that no live process maps** before boot — the liveness check is a `/proc/[pid]/maps` scan, so a second engine's live region on the same host is left alone. Your restart-loop scenario (engine dies, `restart: unless-stopped` re-enters, ghost blocks `madvise`, loop) becomes one deleted file and a log line. One honest caveat for your exact topology: inside a container with `ipc: host`, the scan can only see the container's own processes, so if you run **several engines sharing the host's `/dev/shm` across namespaces**, set `VLLM_OFFLOAD_KEEP_SHM=1` and keep your cron watchdog — the cleanup cannot ...

This is the best piece of debugging this tracker has seen — five reboots on a card that wedges per attempt, a bit-exact CPU reimplementation as the control, and the discipline to notice the faulting frame *drifting* and conclude the code in the traceback is a bystander. "When vanilla elementwise kernels fault, the page mappings under them are guilty" is the sentence the next GA100 owner needs, and it is now **gotcha 41** on main (`60daef8`), with your patch linked and the `expandable_segments` independent-trigger warning kept next to it. Your 170HX numbers are in the README's new "Results from other hardware" section, method-flagged (133.7 on the shipped target — the first sm80 datapoint, level with the 3090 — and 97.8 on your w8a16 with the workaround).

## Taking you up on the candidate patch

Your in-tree diagnosis points at a fix I can actually build and half-test from here: keep the repack's Python-side intermediates small — chunked unpack/pack over bounded column slices, int32 st...

Two follow-ups from running this in anger.

**Edge 6 escalated from ops-hygiene to a production outage.** The engine died once (host OOM, most likely — nothing in the engine log), and then `restart: unless-stopped` did the rest: every automatic restart leaves the previous incarnation's mmap file behind, the next boot hits `OSError: [Errno 14] Bad address` in `shared_offload_region.py` `madvise`, and the container loops. We found it **70 restarts and ~3 hours later** with 70 ghost files in `/dev/shm` (sparse, so they fit — until reservation fails). A boot-time unlink of stale `vllm_offload_*.mmap` not owned by a live engine would turn this whole class into a non-event; we've deployed a cron watchdog as a stopgap (detect `restarting` + `Bad address` → wipe ghosts → restart).

**KVarN KV (`kvarn_k4v2_g128`) + OffloadingConnector: store path works, load path never hits.** Same protocol as the bf16 runs in the report: eviction traffic flows (41 GB GPU→CPU over a full ladder+pyramid program ...

**Root cause found and worked around on our 170HX — the IMA is sm80 VMM corruption under allocation churn, not a kernel pointer bug.** Full story below; patch linked at the end.

## What actually faults

With your `7db2d1f` fail-fast and `expandable_segments:False`, the w8a16 target + W4A16 DFlash2 drafter still Xid-31'd during `gptq_marlin_repack` — but the faulting frame **drifted between runs** (once `empty_cache`, once a `.to(int32)` cast, once mid-repack), always with `Xid 31 FAULT_INFO_TYPE_REGION_VIOLATION … VIRT_WRITE` at a different VA. Classic async-fault signature: the Python traceback is a bystander.

Controlled experiments (5 reboots, sadly the card wedges per attempt):

| Control | Result |
|---|---|
| Bit-exact torch reimplementation of the repack, real weights, **sm86 (3090)**, `compute-sanitizer` | **0 errors** |
| Full engine-faithful drafter load on **sm80**, 10 GB allocator pressure | clean |
| Same chain on **sm80**, production occupancy (~27 GB resident) | **Xid 3...