ollaya-dev/ollaya

@emnlmn, thanks for the numbers and the v0.5.0 notes. Plan:

- **Prompt.** We port v0.5.0 (`PROMPT_VERSION` 6).
- **Quantization.** The default tag will be Q8_0: our rule is Q8_0 wherever it fits, and at 2.7 GB it fits everywhere. Since Q4_K_M measures the same, it can be a second tag for small machines.
- **Goldens.** We write our own from stock `llama-server` (pinned b11146) on each device we support, so the gate is reproducible on our hardware. What helps most from your side is an independent check of the prompt port. Here is our shared request set as JSONL, one `{"id", "state", "questions"}` per line, the requests exactly as a client sends them: [ollaya-shared-requests.jsonl](https://gist.github.com/cobanov/dffe49d26ce328aa97716c3085249fef) (121 requests: the edge cases and 40 typed-decisions rows, sha256 `c95b85a2…24c09e30`). If you can run `snap export-prompts` on it and post the output, that is enough: for each request, the rendered prompt per question, the letter slots and, if ...

@cobanov thanks, this plan works for me. Updates since your reply:

- **Quantization**: we measured Q8_0 against Q4_K_M on typed-decisions (snap
0.5.0, M1 Max): 0.655 vs 0.654 accuracy, so Q4_K_M loses nothing measurable.
Q4_K_M is our default for size; Q8_0 is equivalent, use whichever fits your
policy.
- **Prompt:** v0.5.0 is out (brew and release assets), `PROMPT_VERSION` 6. It
renders the yes/no outcome descriptions as `Yes:` / `No:` lines and falls
back to the option key for null descriptions; ordinary requests are
byte-identical to v5. Accuracy on typed-decisions is 0.655, from 0.643 on
v0.4.0 (v5). We'd suggest v0.5.0 as the porting target, with v0.4.0 (v5) as
the fallback if you've already started. I've updated the numbers in the
issue body and the reproduce steps.
- **Goldens:** `snap export-prompts` dumps the rendered prompts and letter slots
of a JSONL set. We can add the token ids and the stock llama-server logits
in whatever format you prefer. Could y...

@emnlmn, thanks, this is a clear proposal and it fits the GGUF path well. Like Cygnet, it reads letter logits after one prompt, so it is one forward pass per question. We would like to add it.

How Ollaya would bring it in:

1. **Port the prompt.** `src/prompts.rs` at v0.4.0 (`PROMPT_VERSION` 5) goes into a Python reference and the Rust layout. Both are checked byte for byte against snap's own prompts on our shared request set, which includes the edge cases: control tokens in user text, long JSON, and more options than letters.
2. **Gate on parity.** Goldens come from stock `llama-server` of our pinned llama.cpp build (b11146) on the device under test. The gate is identical prompt ids, the same decisions, and option logits within 1e-3.
3. **Measure.** Typed-decisions accuracy and latency through Ollaya's own runtime, published next to the other models on ollaya.dev/results.

Your offer of golden outputs is welcome. The most useful form would be, for each request, the prompt token ids s...

@solarpush, thanks for the proposal and the follow-up on keeping the core lean. Sorry it took a while to answer.

We have decided not to put training into Ollaya itself. Ollaya runs authors' models unmodified and gates each one on parity against the author's own code. A training loop is a different product, with its own dependencies, hardware matrix and support load, even as a separate binary.

What fits is the other half: making it easy to run a model you trained elsewhere. `kev` already works this way: Jared Palmer's LoRA adapters run on the unmodified Qwen3.5 base. An import path for a LoRA adapter (plus its head or label tokens and its calibration) on a base Ollaya already runs would let any training tool (yours, Unsloth, Axolotl) end in a model Ollaya can load and check.

If you would like to build the training side as its own project that targets such a format, we would be glad to link it from the docs, and the import format is something we can design together in this issue.

Thanks @drsect0r and @Xe, that helps a lot. Here's the parity check for `winnow:e4b` on llama.cpp's CUDA backend on a GB10. It's the same gate every device passes: Ollaya's runner against a stock `llama-server` of the pinned build (b11146), on the same GPU, with every decision identical and option logits within 1e-3.

ggml-org's `ubuntu-cuda-13.4-arm64` build of b11146 includes `sm_121a` kernels, so GB10 is covered. I ran these exact steps on x86-64 with the x86 archives (505/505, logits within 1.3e-5).

**You need:** `git`, `curl`, `python3` (standard library only), a Rust toolchain (`curl https://sh.rustup.rs -sSf | sh`), the CUDA 13 runtime that DGX OS ships (`libcudart.so.13`, `libcublas.so.13`), and about 9 GB of disk.

```sh
git clone https://github.com/ollaya-dev/ollaya && cd ollaya
T=$PWD/gb10 && mkdir -p $T/model

# 1. llama.cpp b11146 from ggml-org, sha256-pinned: the CPU libraries and the CUDA backend
sh scripts/llama-cpp.sh linux-arm64 $T/lib/llama
sh scripts/llama-cpp.sh l...

I also have a DGX Spark (aarch64, GB10) and would be happy to help test the ARM64 CUDA pack if needed. Share the build and parity-check commands, and I can run them and report the results and logs.

To follow up on this proposal, here are some thoughts on how the architecture could be cleanly designed **without bloating the core Ollaya runtime**:

### 1. Absolute Rule: Zero Bloat in Core Ollaya (`ollaya` & `ollaya-runner`)

The core inference daemon must stay 100% lightweight, pure, and dependency-free:
- **No heavy autograd frameworks (libtorch, PyTorch, CUDA/ROCm compilers)** linked into the release binaries.
- Users who only run inference download the exact same compact binary and packages as today.

### 2. Proposed Decoupled Architecture

We can keep training completely out-of-process, mirroring the clean pattern already established by `convert/` and `scripts/llama-cpp.sh`:

* **Option A (Thin CLI Dispatcher + UV):**
- The `ollaya` CLI only provides a thin command parser for `ollaya train`.
- It delegates out-of-process to an isolated, UV-managed Python toolchain (`convert/ollaya_train/`).
- When a user runs `ollaya train`, `uv` resolves the training dependencies (PyTorc...

Good question. There are two engines, and aarch64 CUDA differs between them:

- **GGUF models on llama.cpp** (`winnow`, `jevk5`): llama.cpp b11146, the build Ollaya pins, ships `ubuntu-cuda-13.4-arm64`, and its CUDA backend can go into a linux-arm64 GPU pack the same way as on x86-64 and Windows (#23). This is the realistic first step, and it covers `winnow:e4b`, the recommended model.
- **ONNX models** (`laya`, `decider`, `kev`, …): Microsoft publishes ONNX Runtime's CUDA build only for x86-64, and linux-aarch64 is CPU-only. GPU support there would mean building ONNX Runtime ourselves for sm_121 (GB10), which is a bigger job.

The blocker is verification. Every device path has to pass the parity gate on real hardware before it ships, and I don't have a GB10. If I prepare a linux-arm64 CUDA pack for the llama.cpp models, would you be up for running the parity check on your Spark? It's one command against a published fixture, and I'd post the exact steps. Leaving this open to track it.