Janas streams experts off SSD to outrun llama.cpp on local MoE models

Janas is an experimental inference engine written in C, with no dependencies beyond the C library, that runs mixture-of-experts models too big to fit in RAM. The approach is familiar by now: a MoE model only fires a few experts per token, so Janas keeps the weights on an NVMe SSD and streams in just the experts each token needs, caching the ones that recur in whatever memory is free. What is new is the speed. On a single laptop with a Core Ultra 9 and 32 GB of RAM, the project reports Qwen3-Next-80B reading at 62 tokens per second against llama.cpp's 16, and writing at 24 against 2.8. It claims to beat llama.cpp on all 13 models it compared.

Some of that comes from reading the disk at full speed with O_DIRECT, some from speculative decoding, where a small draft proposes tokens the full model then checks. Drafts were accepted 74 to 89 percent of the time on English prose, though only about a third on Italian for one small model. The author is blunt about the cost: Janas fills free memory with its cache and hammers the disk, so on a machine with less headroom the desktop can swap heavily or freeze, and a GPU run once triggered a driver reset. Every number comes from one machine, and the author calls it "an experiment, not a product."

Why it matters

If you run open models locally, this points at a faster path than buying the RAM you do not have: fast storage plus smart streaming. The caveats are real, so treat the throughput claims as a reason to benchmark it on your own hardware, not a promise.