BaseRT

The discussion's focused heavily on prefill vs. decode, but there's a third axis worth thinking about for Apple Silicon specifically: unified memory's effect on how many models you can keep resident simultaneously. llama.cpp and MLX both require fairly explicit load/unload, and on a machine where you might want a small coding model, a larger reasoning model, and an embedding model all warm at once, the round-trip cost of loading from disk breaks the local-first experience more than raw token throughput does. Is BaseRT doing anything with model-weight sharing or partial preloading, or is that a separate problem from what you've optimized?

On-device inference is exactly the direction I keep hoping more indie apps can go — no per-token bill and privacy by default is a big deal. Is BaseRT meant to be embedded inside a shipping Mac/iOS app, or is it more of a dev/CLI runtime for now? That'd decide whether I could actually bundle it into my own app.

Is it also on huggingface? I don't want to install it via bash command.

6.4x over llama.cpp is a big claim for a field this optimized — where does the speedup actually come from? Custom Metal kernels, better KV-cache layout, speculative decoding, or quantization formats? And does the gap hold at batch-1 decode on long contexts, or is it mostly a prefill win? Genuinely curious — we run local models for dev work and llama.cpp has been the default for so long that a 6x claim deserves a look.

6.4x faster than llama.cpp is a big claim — what's the tradeoff? Curious whether it's fast enough to run a real-time voice pipeline on-device (where every 100ms of latency is felt), or if the win is mainly on batch/throughput. Running local with no token cost + full privacy is a huge deal for small-business use cases. Following.

Very interesting work. It feels like we're finally seeing software catch up to what Apple Silicon has been capable of for a while.One thing I'm curious about: if an application needs to switch between different models depending on latency or quality requirements, does BaseRT help manage that orchestration, or is that intentionally left to the application layer?

the no-per-token part matters as much as the speed imo, it's what makes always-on background agents affordable to run. is that the direction you're headed?

the prefill numbers are impressive, but prefill isn't usually where I feel the wait on a laptop - it's token-by-token decode speed during a long generation. is the decode-side speedup in the same range as prefill, or is that gap smaller since decode is more memory-bandwidth bound than compute bound?

@lukas_base Benchmarks you posted seems to be prefill numbers. do you have decode or token generation throughput comparisons against llama.cpp and MLX too, or is most of the gain concentrated in prompt processing rather than generation speed? Thanks!

@lukas_base Just a quick question how does it compare to LiteRT? It is no secret that MLX and llama.cpp are falling behind but we already have a great architecture from Google.

Installed it on my M2 and pulled a 7B model in under a minute, felt snappy even on battery. Nice to skip the per-token bill without cooking my laptop.

curious how it handles model swapping on the fly, like dropping into a bigger LLM only when a prompt clearly needs it. A simple toggle or auto-routing between something like Llama 3 8B and 70B based on the task would be a really nice touch for battery life on MacBooks.

Took a few minutes to set up on my M2 and the speed genuinely surprised me compared to other runtimes I've tried. Love that everything stays on device, no tokens flying off to some server.

One thing I'd love to see is a simple built-in model downloader so I can grab and switch between popular open weights from the CLI without hunting for GGUF files manually. That would make testing new models way smoother.

Love how simple the install is and the speed on my M2 is genuinely impressive. One thing that would make this even better is a built-in model benchmark tool so you can see tokens/sec for different models on your specific hardware right from the CLI.