SaaS Metrics
Gemini Executive Synthesis
Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains.
Technical Positioning
Optimizing multi-GPU inference performance for Qwen3.8-27B on cost-effective hardware (dual RTX 3090s) to achieve significant throughput improvements (+16-35%) across various concurrency levels and speculative decoding methods (MTP, DFlash2), while resolving memory allocation inefficiencies.
SaaS Insight & Market Implications
This detailed A/B test confirms significant performance gains (16-35% throughput increase) by scaling Qwen3.8-27B inference from single to dual RTX 3090s using Tensor Parallelism. The identification and resolution of the KV_MEM pinning issue, which previously stranded 8 GiB per card, directly improves resource utilization and validates the multi-GPU scaling strategy. This demonstrates a clear path for B2B customers to enhance inference capacity and throughput on readily available hardware. The collaborative, data-driven approach to identifying and fixing performance bottlenecks, even in complex multi-GPU setups, reinforces the value of open-source contributions and responsive development for critical infrastructure components like vLLM.
Proprietary Technical Taxonomy
Raw Developer Origin & Technical Request
GitHub Issue
Aug 25, 2026
Repo: syv-ai/qwen38-27b-rtx3090
Dual RTX 3090 (TP=2) vs single: a controlled A/B on your harness — +16-35%, DFlash2 residency, and the KV_MEM pin
Hi, and first of all — thank you for this repo. The care in it is unusual: the gotchas file alone
saved us a day, and having every claim tied to a harness you ship means an outside report like this
one can actually be compared to yours instead of talking past it. We really appreciate the work.
We have a dual-3090 box and wanted to share the tests we were able to run on it. The README notes
that multi-GPU "goes through untouched … every multi-GPU number in the issues is a user report
rather than something I have reproduced", and #7 invites numbers, so we hope this is useful rather
than noise. We ran it as a **controlled A/B** — same box, same install, same
`bench/run_benchmarks.sh single`, same session, with `--tensor-parallel-size 2` as the only
variable — because what seemed missing wasn't a dual-GPU number but the **delta**: how much the
second card is actually worth on this stack, measured against itself.
Two of the findings touch things you documented deliberately (the DFlash2 single-stream guidance
and the pinned `KV_MEM`), so we've tried to show our work and flag exactly what we did and didn't
measure. If we've misread either, we'd rather be corrected than have the numbers stand. Happy to
re-run anything, share raw logs, or test a patch on this hardware.
---
## TL;DR
1. **The second 3090 pays off across the whole concurrency range**, not just in capacity:
**+16% to +32%** with MTP, **+19% to +35%** with DFlash2 (greedy, decode tok/s). Best result:
**161.6 tok...
Developer Debate & Comments
This is really useful information. I'm also running a very similar setup and had planned to provide some numbers this weekend but the information you've provided is so much more in depth. Also, your numbers are far better than mine so I will spend some time comparing configurations and report back if I can tweak things and achieve the same results. To give a very rough idea of what I'm seeing: 10920x, 128GB quad channel DDR4, 2 x 3090 over dual 16x PCIE3 with p2p (but no nvlink). Caps out at around 122 tok/sec with ~300k context running CTX=fast.
I've changed my configuration to roughly match your suggestions and (warm) benched at 250W and 275W. My numbers are significantly better than they were though a little lower than yours at C1. I suspect there may be some thermal throttling going on as I've not had this setup for long and probably need to tweak things. I'll also add that I'm fairly new to running local models and vLLM in general so please feel free to point out any rookie errors. Setup: - Ubuntu 26.04 - Docker deployment - 10920x, 128GB DDR4, Dual 3090 (EVGA Super FTW3), 16x PCIE3 for both cards, no nvlink, p2p driver patch - Commit: https://github.com/syv-ai/qwen38-27b-rtx3090/commit/60daef8255b6757d9791955a44bce27df1658ea6 My new .env: ```env VLLM_API_KEY=REMOVED CUDA_VISIBLE_DEVICES=0,1 GPU_UTIL=0.93 CTX=fast MAX_LEN=294912 MAX_SEQS=8 SPEC=dflash2 PREFIX_CACHE=1 VISION=1 KV_MEM= DFLASH_TOKENS=7 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 EXTRA_ARGS='--tensor-parallel-size 2 --disable-custom-all-reduce --hf-overrides {"t...
This is the best-run field report this repo has received — a genuine controlled A/B, spread stated, negative results kept, and the two findings that touch deliberate calibrations flagged instead of papered over. Both of those findings are correct, and both are now fixed on main. Point by point, then what I owe you, then @dpcartwright's T=15 result, which I think is the most interesting datapoint in the thread. ## 4.3 — the KV_MEM pin: you are right, and the launcher now skips it at TP>1 The pin is a single-card constant (the 24 GiB transient margin is sharp — 396 MiB dead / 436 MiB fine, gotcha 39), and vLLM applies `--kv-cache-memory` per worker, so at TP=2 it strands ~8 GiB per card exactly as you measured. As of `60921ed` the launcher detects `--tensor-parallel-size`/`-tp` in `EXTRA_ARGS` and, unless you exported a `KV_MEM` yourself, sizes the pool from `GPU_UTIL` with a note — your suggested fix, taken as suggested. Your unpin A/B (302,223 tokens for deltas inside spread, boots c...
No problem. I kept everything else the same and only made the .env changes you specified. I know you said C1 was all that is needed but I just ran the same bench script as it's nice for me to be able to compare in full. Still running the same commit from the previous run: https://github.com/syv-ai/qwen38-27b-rtx3090/commit/60daef8255b6757d9791955a44bce27df1658ea6 **LOOKUP=0 DFLASH_TOKENS=15 (otherwise exact same as previous .env)** | Cohort | Temp | Decode (tok/s) | e2e (tok/s) | Tokens/Step | Mean TTFT (ms) | |---|---|---|---|---|---| | C1 | default | 134.2 | 130.56 | 3.45 | 135.77 | | C2 | default | 178.7 | 161.62 | 3.19 | 188.14 | | C4 | default | 240.0 | 203.10 | 3.43 | 305.49 | | C8 | default | 257.2 | 205.92 | 3.13 | 721.15 | | C1 | greedy (T=0) | 130.5 | 126.02 | 3.30 | 140.17 | | C2 | greedy (T=0) | 191.2 | 166.35 | 3.32 | 185.53 | | C4 | greedy (T=0) | 243.8 | 208.12 | 3.45 | 288.03 | | C8 | greedy (T=0) | 289.2 | 231.50 | 3.49 | 730.25 | For this one I'm posting the ful...
Adjacent Repository Pain Points
Other highly discussed features and pain points extracted from syv-ai/qwen38-27b-rtx3090.
Extracted Positioning
Bug tracking and resolution for KVarN, DFlash2, and CTX=huge configurations, specifically addressing output corruption, engine crashes, and KV cache memory limitations.
Ensuring correctness and stability for Qwen3.8-27B inference with advanced speculative decoding (MTP, DFlash2) and extended context lengths (CTX=huge) on RTX 3090, by systematically identifying and fixing critical bugs related to prefill handling, sliding window guards, and KV cache allocation.
Top Replies
Two follow-ups to [my comment on ](https://github.com/syv-ai/qwen38-27b-rtx3090/pull/13#issuecomment-5379687913), moved here since that PR is closed: the MTP result re-run to your fresh-server stan...
**Update — two of the three open items are closed, both verified on hardware.** Pushed in `82bd62d`. ## `DFLASH_TOKENS=15` does not boot at 240k → **fixed** My first fix was aimed at the wrong thin...
Our comments crossed — mine went up six minutes before yours, so this is about one line in your update rather than the conclusion. You landed on `mtp` keeping PIECEWISE, which is where I would land...
Extracted Positioning
Compatibility and stability of w8a16 (GPTQ int8) quantized models with various linear kernels (Marlin, Exllama, Conch) and speculative decoding (DFlash2) on NVIDIA CMP 170HX, specifically addressing CUDA illegal memory access (IMA) and dtype mismatches.
Achieving robust, high-performance inference for GPTQ int8 quantized Qwen3.8-27B models on specialized hardware (CMP 170HX) by ensuring compatibility across different linear kernels and resolving critical dtype-related failures and memory corruption issues.
Top Replies
This is an unusually good report — five attempts with the blame isolated by controls, and a working-vs-broken vLLM version to bisect against. Taking your two asks in reverse order, because I can ac...
**Correcting myself on row 5.** I said I had taken your rejection-sampler assert at face value as a second bf16 assumption and had not read that kernel. I have now, and **it is not one.** `rejectio...
**Root cause found and worked around on our 170HX — the IMA is sm80 VMM corruption under allocation churn, not a kernel pointer bug.** Full story below; patch linked at the end. ## What actually fa...
Extracted Positioning
Functionality and performance of OffloadingConnector (KV eviction to RAM/disk) for extending KV cache capacity on the Qwen3.8-27B stack.
Enabling deployment of Qwen3.8-27B with extended context lengths beyond GPU VRAM limits by leveraging system RAM/disk for KV cache, even in virtualized environments (Proxmox VM with GPU passthrough).
Top Replies
Two follow-ups from running this in anger. **Edge 6 escalated from ops-hygiene to a production outage.** The engine died once (host OOM, most likely — nothing in the engine log), and then `restart:...
Answers to all three open questions, one shipped fix for the outage class, and one debt I am keeping. Your follow-up comment upgraded this report from useful to load-bearing — a 70-restart ghost lo...
The traced cell I owed you is done — your "store path works, load path never hits" is **reproduced on this card, mechanism found, and it is neither unliftable-by-design nor a hash miss**. It's an a...
Extracted Positioning
Stability of vLLM EngineCore with flashinfer attention backend and MTP speculative decoding, specifically concerning CUDA illegal memory access errors during long context processing.
Ensuring robust and error-free inference for Qwen3.8-27B (W4A16 AutoRound) on RTX 3090, particularly with advanced features like flashinfer and MTP, even with long contexts (~28k tokens).
Top Replies
Good report — the dump_input capture and the dmesg line are exactly what makes this analyzable from a distance. One red herring to clear first, then where the real suspicion points, then what I wou...
## Follow-up: second occurrence on 2026-08-27, same fault address The same crash recurred today (2026-08-27 14:03:48 local, after ~40h uptime, docker restart policy auto-recovered). **New data poin...
## Test result: option 1 (`--attention-backend FLASH_ATTN`) is hard-invalid on the fp8-KV tier As suggested, I added `--attention-backend FLASH_ATTN` to the `CTX=long` launch (everything else uncha...
Extracted Positioning
System RAM requirements for loading and running Qwen3.8-27B with vLLM, specifically addressing limitations with 16GB main system RAM.
Enabling deployment of Qwen3.8-27B on systems with constrained main memory, potentially through optimized loading strategies or reduced memory footprint.
Top Replies
Honest framing first: 16 GB of host RAM (10 GiB actually free) is below anything this stack has been run on, so what follows is diagnosis plus the two levers that exist — not a tested recipe. It ma...
> system ram is 16GB, using iGPU Well, your iGPU is using system RAM; what OS & have you tried to reduce background processes? Close as many web browser tabs as you can, & you might have to use thi...
Frequently Asked Questions
Market intelligence mapped to Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains..
What problem does Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains. solve?
Based on our AI analysis of the original developer request, its primary technical positioning is: Optimizing multi-GPU inference performance for Qwen3.8-27B on cost-effective hardware (dual RTX 3090s) to achieve significant throughput improvements (+16-35%) across various concurrency levels and speculative decoding methods (MTP, DFlash2), while resolving memory allocation inefficiencies.
How is the developer community reacting to Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains.?
Yes, we have tracked 4 direct responses and active debates regarding this specific topic originating from GitHub Issue.
What architecture is tied to Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains.?
Our proprietary extraction maps Performance scaling of Qwen3.8-27B with vLLM using dual RTX 3090s (Tensor Parallelism TP=2) compared to a single card, specifically addressing the impact of KV_MEM pinning and overall throughput gains. to adjacent architectural concepts including Dual RTX 3090 (TP=2), Tensor Parallelism, controlled A/B, MTP.
Engagement Signals
Cross-Market Term Frequency
Quantifies the cross-market adoption of foundational terms like MTP and Dual RTX 3090 (TP=2) by tracking occurrence frequency across active SaaS architectures and enterprise developer debates.