Niko1221/Strata

The NVMe tier is part of the unified plan in #57 now (spill on eviction on the shared snapshot core). Closing in favour of #57.

@j-luwierski's #84 implements none/linear/YaRN scaling; let's continue there (one implementation). The open points are in its description; it also needs a perplexity check inside 262K so the default path is unchanged.

Closing as explained in the thread: the Coder keeps half the experts but stores each one larger (IQ1_M packs them like IQ3_S), so fewer of its experts fit in VRAM and decoding is a bit slower than Q2_0; its advantage is fitting in 32 GB of RAM. Thanks for the question!

Closing as fixed, as @dandandelion confirmed on 0.1.19 (a verify-window read past its buffer, fixed in 0.1.19). Thanks for the detailed logs! If it ever comes back, please reopen with `dmesg | grep -i xid` and the engine log.

Engine 0.1.22 (released today) runs one model across two or three NVIDIA GPUs: `START-HERE.bat` / `./setup.sh` lists your cards, says which ones Strata can use, and asks which to use (the two best together recommended); `--gpus 0,1` also works at any start. The model's layers are split across the cards (pipeline parallelism, see docs/MULTI_GPU.md); tensor parallelism isn't planned. Thanks for asking - closing; please open a new issue if something doesn't work on your cards.

Thanks to both of you. The crash was fixed by adding `--vram-reserve-mib 1200` and I updated to 0.1.19, re-setup (which removed the `--vram-reserve-mib 1200` line), and it's not crashing at long generation anymore, in fact, it seems to be alot faster.

```console
[strata] answering: 45932 of max 131001 tokens, 48.4 tok/s, 949 s
[strata] answering: 45963 of max 131001 tokens, 48.4 tok/s, 951 s
[strata] done: 45968 tokens in 951 s (48.4 tok/s) (stop, cancel=False), expert cache 73.2% hit
```
Prefill is around ~880 t/s at 46k which seems to be also faster.

I think this can be closed now. Thank you @Marshal-Douglas-3rd @Niko1221

I tested all you suggestions.
--pool-workers 11
--kv-resident 64072
Windows large pages activated
Unfortunately, it didn't work very well. On the contrary I went from 40 tps to 35 and it was destoied to 18tps when context began to fill
As I upgrade my RAM to 64Gb now, i'll try better quants.
Thanks much

can you add

"--pool-workers",
"4"

to your args and see what that does? Curious

```
{
"exe": "engine\\strata.exe",
"args": [
"--pack",
"..\\Strata-data\\packs\\iq3_s",
"--native",
"..\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
"--ple-gguf",
"..\\Strata-data\\models\\IQ3_S\\Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
"--expert-profile",
"data\\expert-profile.bin",
"--expert-cache",
"auto",
"--prefill",
"auto",
"--spec",
"4",
"--spec-min-p",
"0.5",
"--mtp",
"..\\Strata-data\\mtp\\rt",
"--max-context",
"65536",
"--kv",
"int8",
"--vision",
"--vram-reserve-mib",
"700",
"--control-vector-scaled",
"data\\experimental-speed-projection\\Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0",
"--control-vector-layer-range",
"4",
"44",
"--cvec-mode",
"project",
"--cvec-dir",
"per-layer"
],
"cwd": ".",
"tokenizer": "..\\Strata-data\\packs\\iq3_s\\tokenizer",
"model_name": "qwen3.8-flash-next-iq3_s",
"log": "strata-iq3_s.log",
"lib_dirs": [
".ve...

Two things I confirmed in the engine that may matter for the image-prompt loops, since image requests only ever take the text path's settings today:

1. A request with an image decodes **greedy, always**. `serve/server.py:279` drops the sampling keys whenever a request carries embeddings, and the engine's `GENI` parser skips everything but `cvec=` (`src/program/generate.cpp:2651`). So `temperature` / `top_p` / `top_k` on a vision request are accepted and then silently discarded — no warning, no `ERR`.
2. That also means the recommended Qwen thinking settings cannot be applied to image requests at all, which is consistent with the loops showing up mainly there.

Not claiming it is the cause — the IQ3_S quant is still an untested suspect — but the fix on the engine side is small: parse and honor the same sampling keys on `GENI`, or answer `ERR` naming the unsupported field so it is never silently ignored.

For reference, I measured the throughput side of the recommended set on the IQ1_M ...

That is the desired outcome since using all the threads ends up with a spin lock and would actually make things slower. You should be seeing higher than 30tps on your machine.

Post your strata-q3_s.json args

Interestingly half the CPU threads (8 of 16) work at 95-99%, whereas the other half are used very little or not at all by the engine.

Post back here if you get any findings please, good or bad

Thanks much for taking time to explaining and offering solutions.
Oh I had 0.1.14 I'll test again

**Why the Coder runs slower (it's not a bug — it's what the pruning trades):**

Routing still picks top-10 experts per layer, so every token makes the same ~480 expert-visits whether the menu is 512 or 256 experts. Pruning doesn't reduce that work. What RCO did is delete experts so it could store the survivors *fatter* — the Coder keeps them at ~3.5–4.5 bits (the down-projections are IQ4_NL) while baseline IQ3_XXS stores all 512 at a uniform ~3.06 bits. So each Coder expert blob is ~1.2–1.7× the bytes of an IQ3_XXS blob (I measured 2.66 MB max blob on the Coder vs 1.51 MB for Q2_0 on the same setup).

On a 12 GB card that bites twice: fatter blobs mean `--expert-cache auto` fits ~2,900 slots instead of ~5,000, so a *smaller fraction* of the layer-expert pairs is resident — and each miss that does happen moves more bytes over PCIe and costs more CPU unpacking in the pool. Decode speed at this scale is basically the miss pipeline, so more-misses-times-fatter wins and the token rate drops...