SaaS Metrics
JustVugg/colibri
Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
View Origin LinkProduct Positioning & Context
AI Executive Synthesis
The developers are positioning 'colibri' as a solution for running immense MoE models on consumer CPUs by streaming experts from disk. The benchmark aims to validate its performance on high-end consumer CPUs (Ryzen 9 9950X) and fast storage (PCIe 5.0 NVMe), demonstrating viability and identifying performance bottlenecks related to RAM, disk, and compute trade-offs.
This issue details a performance benchmark for 'colibri', an engine designed to run 744B MoE models on consumer CPUs by streaming experts from disk. The benchmark, conducted on a Ryzen 9 9950X with PCIe 5.0 NVMe storage and 128GB RAM, yielded a median throughput of 0.28 tokens/second for 64-token generations. This performance indicates significant latency for practical applications, despite leveraging high-end consumer hardware and fast storage. The high RSS (82.15 GB) suggests substantial memory consumption even with disk-streaming. The 56.7% expert hit-rate implies frequent disk I/O, directly impacting throughput. The market implication is that while the architectural approach of disk-streaming MoE experts on CPU-only systems is technically demonstrated, current performance metrics are insufficient for real-time or interactive use cases, limiting its immediate B2B SaaS viability for high-throughput inference. Further optimization is critical to improve token generation rates and reduce effective memory footprint.
Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
Related Ecosystem & Alternatives
Discover adjacent products, open-source repositories, and developer tools sharing similar technical architecture.
Deep-Dive FAQs
What is JustVugg/colibri?
JustVugg/colibri is analyzed by our AI as: The developers are positioning 'colibri' as a solution for running immense MoE models on consumer CPUs by streaming experts from disk. The benchmark aims to validate its performance on high-end consumer CPUs (Ryzen 9 9950X) and fast storage (PCIe 5.0 NVMe), demonstrating viability and identifying performance bottlenecks related to RAM, disk, and compute trade-offs.. It focuses on This issue details a performance benchmark for 'colibri', an engine designed to run 744B MoE models on consumer CPUs by streaming experts from disk...
Where did JustVugg/colibri originate?
Data for JustVugg/colibri was aggregated directly from the GitHub Open Source community ecosystem, representing raw developer and early-adopter sentiment.
When was JustVugg/colibri publicly launched?
The initial public indexing or launch date for JustVugg/colibri within our tracked developer communities was recorded on July 1, 2026.
How popular is JustVugg/colibri?
JustVugg/colibri has achieved measurable traction, logging over 9,975 traction score and facilitating 785 recorded discussions or engagements.
Are there active development issues for JustVugg/colibri?
Yes, we are currently tracking open architectural debates and bug reports for this project on GitHub. There are currently 4 active high-priority issues logged recently.
Are there open-source alternatives related to JustVugg/colibri?
Yes, the GitHub ecosystem contains correlated projects. For example, a repository named fikrikarim/parlor shares highly similar architectural descriptions and topics.
How does the creator describe JustVugg/colibri?
The original author or development team describes the product as follows: "Run GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦"
Active Developer Issues (GitHub)
Logged: Jul 10, 2026
Logged: Jul 10, 2026
Logged: Jul 10, 2026
Logged: Jul 9, 2026
Community Voice & Feedback
Good catch — `nvidia-smi` shows nothing because you're running the **CPU-only build**: `setup.sh` builds plain `make`, and the CUDA tier is opt-in at build time. To actually light up the A6000:
```sh
cd c && make CUDA=1 # needs CUDA toolkit under /usr/local/cuda (or CUDA_HOME=...)
COLI_MODEL=... ./coli chat --auto-tier
```
`--auto-tier` applies the plan you already printed (48.5 GB VRAM hot tier on device 0) — but only when the binary is CUDA-enabled; with the CPU binary it correctly falls back, which is what you saw.
Two expectation-setting notes before you burn a day on it:
1. **The VRAM tier promotes experts from the PIN hot-store, and the pin comes from your `.coli_usage` history.** Yours was ~39k selections (~7 GB worth) — so the first CUDA session will only place ~7 GB of the 48 GB tier. It grows every session; matey-0 in #12 is at 426k selections = 46.7 GB pinned. Chat/bench for a few sessions and the tier fills itself.
2. **Measured honestly (#16, fixture): expert-...
```sh
cd c && make CUDA=1 # needs CUDA toolkit under /usr/local/cuda (or CUDA_HOME=...)
COLI_MODEL=... ./coli chat --auto-tier
```
`--auto-tier` applies the plan you already printed (48.5 GB VRAM hot tier on device 0) — but only when the binary is CUDA-enabled; with the CPU binary it correctly falls back, which is what you saw.
Two expectation-setting notes before you burn a day on it:
1. **The VRAM tier promotes experts from the PIN hot-store, and the pin comes from your `.coli_usage` history.** Yours was ~39k selections (~7 GB worth) — so the first CUDA session will only place ~7 GB of the 48 GB tier. It grows every session; matey-0 in #12 is at 426k selections = 46.7 GB pinned. Chat/bench for a few sessions and the tier fills itself.
2. **Measured honestly (#16, fixture): expert-...
np, re pulled.
Please confirm the below before I let it run. `nvidia-smi` doesn't pick up inference.
------------------
❯ ./setup.sh
🐦 colibrì — setup
gcc: 13 · 32 core
OpenMP: ok
compilo (ARCH=native)…
RAM: 131 GB (più RAM = più expert in cache = più veloce)
pronto. Prossimi passi:
./coli build # (gia' fatto)
./coli convert --model /percorso/NVMe/glm52_i4 # genera il modello int4 (ore)
./coli info --model /percorso/NVMe/glm52_i4
./coli chat --model /percorso/NVMe/glm52_i4 --ram
IMPORTANTE: tieni il modello su disco VELOCE (NVMe/ext4), MAI su /mnt/c o rete.
❯ COLI_MODEL=/home/cbroker/Documents/2026/colibri ./coli plan
▄▀▀▀▄ ▄ colibrì v1.0
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ plan · Disk / RAM / VRAM...
Please confirm the below before I let it run. `nvidia-smi` doesn't pick up inference.
------------------
❯ ./setup.sh
🐦 colibrì — setup
gcc: 13 · 32 core
OpenMP: ok
compilo (ARCH=native)…
RAM: 131 GB (più RAM = più expert in cache = più veloce)
pronto. Prossimi passi:
./coli build # (gia' fatto)
./coli convert --model /percorso/NVMe/glm52_i4 # genera il modello int4 (ore)
./coli info --model /percorso/NVMe/glm52_i4
./coli chat --model /percorso/NVMe/glm52_i4 --ram
IMPORTANTE: tieni il modello su disco VELOCE (NVMe/ext4), MAI su /mnt/c o rete.
❯ COLI_MODEL=/home/cbroker/Documents/2026/colibri ./coli plan
▄▀▀▀▄ ▄ colibrì v1.0
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ plan · Disk / RAM / VRAM...
> Of course, I'm here, I always read everything! That's why I'm promoting a Discord server. If there's interest, I'll create one and we can talk there. We can still talk and organize something, maybe a Zoom call or something.
Dude, I really like you—let's get this project started!
Dude, I really like you—let's get this project started!
Of course, I'm here, I always read everything! That's why I'm promoting a Discord server. If there's interest, I'll create one and we can talk there. We can still talk and organize something, maybe a Zoom call or something.
> Great energy — and after [#16](https://github.com/JustVugg/colibri/pull/16) there's no doubt you can drive serious pieces of this. Let's sort the list into what fits the project's soul and what doesn't.
>
> **The one hard no: a Rust rewrite.** colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.
>
> **Everything else is welcome, and most of it already has a seed to grow from:**
>
> * **Tiered VRAM/RAM/disk** — you literally built the VRAM tier in [feat: add tiered CUDA acceleration for routed experts #16](https://github.com/JustVugg/colibri/pull/16). Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
> * **Hotspot analysis** — the learning cache (`.c...
>
> **The one hard no: a Rust rewrite.** colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.
>
> **Everything else is welcome, and most of it already has a seed to grow from:**
>
> * **Tiered VRAM/RAM/disk** — you literally built the VRAM tier in [feat: add tiered CUDA acceleration for routed experts #16](https://github.com/JustVugg/colibri/pull/16). Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
> * **Hotspot analysis** — the learning cache (`.c...
---
## 📊 Community vote: Discord server?
@JustVugg is ready to create a **project Discord** so design discussions like this one can happen in real time (GitHub issues stay the home for bugs, benchmarks and decisions).
**Vote with a reaction on THIS comment:**
- 👍 — yes, I'd join a colibrì Discord
- 👎 — no, keep everything on GitHub
If the 👍 win, the invite link lands in the README.
## 📊 Community vote: Discord server?
@JustVugg is ready to create a **project Discord** so design discussions like this one can happen in real time (GitHub issues stay the home for bugs, benchmarks and decisions).
**Vote with a reaction on THIS comment:**
- 👍 — yes, I'd join a colibrì Discord
- 👎 — no, keep everything on GitHub
If the 👍 win, the invite link lands in the README.
Great energy — and after #16 there's no doubt you can drive serious pieces of this. Let's sort the list into what fits the project's soul and what doesn't.
**The one hard no: a Rust rewrite.** colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.
**Everything else is welcome, and most of it already has a seed to grow from:**
- **Tiered VRAM/RAM/disk** — you literally built the VRAM tier in #16. Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
- **Hotspot analysis** — the learning cache (`.coli_usage` + AUTOPIN with confidence scaling) is exactly this today. Real headroom: expert co-occurrence analysis, disk-layout reordering so co-firing experts read ...
**The one hard no: a Rust rewrite.** colibrì's identity is a single C file you can read in an afternoon — gcc and go, no dependency tree. Performance-wise a rewrite buys nothing here: the hot paths are bounded by disk and memory bandwidth, not by the language. A Rust port as a sibling project elsewhere is of course fine, but the reference engine stays C.
**Everything else is welcome, and most of it already has a seed to grow from:**
- **Tiered VRAM/RAM/disk** — you literally built the VRAM tier in #16. Deepening it (smarter placement, per-expert precision by heat, promotion/demotion at runtime) is a natural continuation.
- **Hotspot analysis** — the learning cache (`.coli_usage` + AUTOPIN with confidence scaling) is exactly this today. Real headroom: expert co-occurrence analysis, disk-layout reordering so co-firing experts read ...
Thanks for the detailed numbers — a 7950X with a 3.85 GB/s buffered NVMe is a great datapoint, and your log exposed the same bug as #12: `[RAM_GB=92.3 auto] cap=8 ok` means the engine was running a 16 GB-machine cache on your 92 GB budget (the cap could only be lowered, never raised).
**That's fixed now — `git pull` and rerun.** The engine auto-raises the expert cache to fill the RAM budget (`CAP_RAISE=0` opts out). On your machine expect the cap to jump from 8 to ~30, and the hit-rate well past the 23–35% you saw.
Two more things from your log:
- `PIN_GB=95` didn't pin 95 GB because the pin can only hold experts your usage history has actually seen — you had 39,848 selections, worth ~7.3 GB. That's the learning cache working as designed: it grows every session (the Framework 13 user in #12 is at 426k selections = 46.7 GB pinned). Just keep using it; `AUTOPIN` handles the budget automatically, no `PIN_GB` needed.
- Your MTP is worth checking: if you downloaded from the mirror early,...
**That's fixed now — `git pull` and rerun.** The engine auto-raises the expert cache to fill the RAM budget (`CAP_RAISE=0` opts out). On your machine expect the cap to jump from 8 to ~30, and the hit-rate well past the 23–35% you saw.
Two more things from your log:
- `PIN_GB=95` didn't pin 95 GB because the pin can only hold experts your usage history has actually seen — you had 39,848 selections, worth ~7.3 GB. That's the learning cache working as designed: it grows every session (the Framework 13 user in #12 is at 426k selections = 46.7 GB pinned). Just keep using it; `AUTOPIN` handles the budget automatically, no `PIN_GB` needed.
- Your MTP is worth checking: if you downloaded from the mirror early,...
Beautiful follow-up — 0.29 → 0.37 tok/s with hit 28% → 66% and speculation finally engaging (52% acceptance, 2.59 tok/fw) is exactly the jump the numbers predicted, and on a one-fan 13" laptop. Your machine is now in the README benchmark table, and your int8-MTP mirror clone is linked in the download section — thank you for both.
Two updates on our side, just pushed:
1. **The cap workaround is no longer needed**: `git pull` and the engine now auto-raises the expert cache to fill your `--ram` budget (it caps at what fits; `CAP_RAISE=0` restores the old behavior). With `--ram 110` you should land around cap≈32 automatically — but if it picks a different value than your manual 32, numbers from a rerun would be interesting.
2. **`PILOT=1` (experimental, just landed) was practically designed for your machine.** Your profile is ~45% disk / ~45% matmul — the router-lookahead prefetch predicts the *next* layer's experts (71.6% measured recall) and reads them from a dedicated I/O thread whil...
Two updates on our side, just pushed:
1. **The cap workaround is no longer needed**: `git pull` and the engine now auto-raises the expert cache to fill your `--ram` budget (it caps at what fits; `CAP_RAISE=0` restores the old behavior). With `--ram 110` you should land around cap≈32 automatically — but if it picks a different value than your manual 32, numbers from a rerun would be interesting.
2. **`PILOT=1` (experimental, just landed) was practically designed for your machine.** Your profile is ~45% disk / ~45% matmul — the router-lookahead prefetch predicts the *next* layer's experts (71.6% measured recall) and reads them from a dedicated I/O thread whil...
I switched to the int8 MTP head(s). Didn't disable ZSTD for this directory, because BTRFS is probably smart enough to—after not being able to compress the first X amount of bytes—just not do anything. I changed my flag to add --cap 32, and ran it 6 times in total.
This was the sixth time:
----
~/Development/colibri/c main +3 !1 ❯ ./coli run "Explain how a hash map works." --model /home/mateogrgic/Development/colibri/c/GLM-5.2-colibri-int4 --ram 110 --ngen 150 --cap 32
▄▀▀▀▄ ▄ colibrì v1.0
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ run
▀
──────────────────────────────────────────────────────────
== Motore C GLM (glm_moe_dsa), cache=32 expert/layer | expert@8-bit densa@8-bit | idot: avx512-vnni ==
caricato in 3.80s | densa residente: 9912.75 MB | layers=78 experts=256 | MTP ATTIVA (draft=3)
[MTP] attiva: decodifica speculativa nativa (draft=3)
[USAGE] storia expert:...
This was the sixth time:
----
~/Development/colibri/c main +3 !1 ❯ ./coli run "Explain how a hash map works." --model /home/mateogrgic/Development/colibri/c/GLM-5.2-colibri-int4 --ram 110 --ngen 150 --cap 32
▄▀▀▀▄ ▄ colibrì v1.0
▄▄▄▄▀▀▀▀▄▀▀ piccolo motore, modello immenso
▀▀▀▀▀▀▀ GLM-5.2 · 744B MoE · int4 · streaming CPU
▀▀▀▀ run
▀
──────────────────────────────────────────────────────────
== Motore C GLM (glm_moe_dsa), cache=32 expert/layer | expert@8-bit densa@8-bit | idot: avx512-vnni ==
caricato in 3.80s | densa residente: 9912.75 MB | layers=78 experts=256 | MTP ATTIVA (draft=3)
[MTP] attiva: decodifica speculativa nativa (draft=3)
[USAGE] storia expert:...
Fantastic datapoint — thank you! Your profile tells a very clear story, and it also uncovered a real bug on our side.
**1. Your expert cache is capped at 8/layer — you're using ~22 GB of your 110 GB budget.**
`coli run`'s default is `--cap 8`, and the engine only ever *lowers* the cap to fit RAM, never raises it (that's the bug — auto-raise fix in progress). On 128 GB you can hold a much bigger LRU right now:
```
./coli run "..." --model ... --ram 110 --cap 32
```
(the engine trims it automatically if it doesn't fit). Expect the hit-rate to climb well past 28% and expert-disk time to drop with it.
**2. MTP acceptance 0% looks exactly like the int4 draft head (#8).**
If you downloaded from the HF mirror early on, your MTP head may be the int4 one — unusable (0–4% acceptance, which is what you see). Check:
```
ls -l /out-mtp-*
# int8 head (good): 3527131672 / 5366238584 / 1065950496
# int4 head (bad): 1765523544 / 2686077736 / 536747200
```
If yours match the int4 sizes, r...
**1. Your expert cache is capped at 8/layer — you're using ~22 GB of your 110 GB budget.**
`coli run`'s default is `--cap 8`, and the engine only ever *lowers* the cap to fit RAM, never raises it (that's the bug — auto-raise fix in progress). On 128 GB you can hold a much bigger LRU right now:
```
./coli run "..." --model ... --ram 110 --cap 32
```
(the engine trims it automatically if it doesn't fit). Expect the hit-rate to climb well past 28% and expert-disk time to drop with it.
**2. MTP acceptance 0% looks exactly like the int4 draft head (#8).**
If you downloaded from the HF mirror early on, your MTP head may be the int4 one — unusable (0–4% acceptance, which is what you see). Check:
```
ls -l /out-mtp-*
# int8 head (good): 3527131672 / 5366238584 / 1065950496
# int4 head (bad): 1765523544 / 2686077736 / 536747200
```
If yours match the int4 sizes, r...
Discovery Source
GitHub Open Source Aggregated via automated community intelligence tracking.
Tech Stack Dependencies
No direct open-source NPM package mentions detected in the product documentation.
Media Tractions & Mentions
No mainstream media stories specifically mentioning this product name have been intercepted yet.
Deep Research & Science
No direct peer-reviewed scientific literature matched with this product's architecture.