A 4-Bit Model That Beats Its Full-Precision Original

Compressing a model almost always costs accuracy, so the interesting claim from Multiverse Computing is that their compressed version can score higher than the full-precision model it came from. Their method, Quantization-Aware Healing, changes what the compressed model learns from. Instead of distilling from an already-degraded checkpoint, it distills from the original full-size, full-precision model held frozen as the teacher, using a KL-divergence loss on the output logits and chunked computation to handle contexts up to 32k tokens.

The test case is GPT-OSS 120B compressed to 60B parameters and quantized to MXFP4, a 4-bit format. Against the bfloat16 recovered version it wins on 7 of 9 benchmarks, including a 7.4-point gain on long-context reasoning (AA-LCR) and 5.6 points on math (AIME 2025). Training is also cheaper to run: the method reaches peak performance in about 100 steps where a competing quantization-aware training approach needs roughly 700, and it stays stable afterward instead of suffering a 19-point collapse. The result is a model at about a quarter of the memory with faster inference that still edges out its uncompressed source, which inverts the usual tradeoff. The write-up on Hugging Face lays out the loss setup and the full benchmark table.

Why it matters

If you deploy open models and fight memory limits, this is a recipe to try rather than a product to buy: pick the frozen full-precision teacher, not the degraded checkpoint, when you heal a quantized model. The reported speed of convergence also makes the approach cheap enough to test on your own weights before committing.