Tim Dettmers bets academic labs can run frontier AI on modest hardware

Tim Dettmers, known for his work on model quantization, used his lab's "open source week" to make a larger argument: the next wave of research will come from academic labs on modest hardware, not only from well-funded frontier companies. His case rests on making large models cheap to run. His team's inference framework runs a quantized Qwen 3.6 35B model at 450 tokens per second at 1.5 bits per weight, roughly a 90% cut in memory against half precision. A 125B model, Qwen 3.8 Flash Next, fits on a single 24GB GPU, and DeepSeek V4.1 at 550B runs on standard AMD or Nvidia cards, or on a MacBook with 128GB of RAM.

The release also includes an auto-compaction technique the team calls CliffCompaction, which lets agent sessions run for millions of tokens. Dettmers writes that some of his own sessions have passed a hundred million tokens, and reports it cut costs by about half, with one deployment measuring a 45% drop in a company's total AI budget. Alongside these he is releasing an autonomous research system that runs fully offline, which he says beat Sakana AI and Google's ScientistOne on his tests and solved a bioinformatics problem in about two hours, producing four results.

The whole set, two open-source projects and four papers, is meant to work together rather than as separate parts.

Why it matters

If you do research or build agents outside a big lab, the wall has usually been raw compute you could not afford. These tools push serious model sizes onto a single GPU or a laptop, so the question shifts from whether you can run a 500B model to what you do with one on your own machine.