Muon, the optimizer everyone switched to, quietly kills neurons

Muon has spread fast as a replacement for AdamW in language model training, so a finding from Tilde Research, highlighted by Jack Clark in Import AI, is worth attention. They report that Muon's update rule slowly kills off neurons: by training step 500, more than one in four neurons in the MLP layers have stopped responding to updates. A quarter of the network is doing nothing.

Tilde built an alternative they call Aurora, described as a leverage-aware optimizer for rectangular matrices. In their runs Aurora reaches a final smoothed loss of 2.26 against Muon's 2.31, and scores ten points higher on MMLU. Those are real gains for a change that only touches the optimizer, not the model or the data.

Clark is careful not to oversell it. He points out that researchers have spent years trying to beat AdamW and that the line of failed attempts is long, so one promising result is not a settled win. The useful takeaway is narrower: a widely adopted optimizer has a measurable defect, and the people who found it also showed you can do better.

Why it matters

If you train models and have moved to Muon for its speed, dead neurons mean you may be paying for capacity you never use. Before your next large run, it is worth measuring how many of your neurons are still alive, and watching whether Aurora's results reproduce outside Tilde's own setup.