Field report: OffloadingConnector (KV eviction to RAM/disk) works on this stack — 8 sharp edges and one open question
syv-ai/qwen38-27b-rtx3090
# Field report: OffloadingConnector (KV eviction to RAM/disk) on top of this stack — it works, with sharp edges
**Setup, so you can discount accordingly:** Ubuntu VM on Proxmox, GPU passthrough, RTX 3090 Ti, image built from `main-7db2d1f`, model = philbert440 AWQ body + our own GPTQ-int4 requant of lm_head/MTP (the "fast2" recipe from your drafter/README applied to a different body), bilingual ru-en draft vocab (18,195 ids), `SPEC=mtp DRAFT_TOKENS=3 PREFIX_CACHE=1`, bf16 KV. All numbers below are VM numbers; we already know from the PR #13 thread that this box pays 2x for PIECEWISE where bare metal pays
View on GitHub ↗
Other Comments / Reviews
SaaS Metrics