Scaling laws look precise, but the way you fit them is fragile

Scaling laws promise something useful: train a few small models, fit a curve, and predict how a much bigger model will do before you spend the compute. Lilian Weng's new explainer walks through how these laws actually behave, and her honest message is that the curve is more delicate than it looks. The headline relationship is simple, training loss falls as a power law in model size, data, and compute, with the rough rule that compute is about six times parameters times tokens. The argument starts when you ask how to split a fixed budget between a bigger model and more data.

She lays out the well-known gap between Kaplan (2020) and Chinchilla (2022). Kaplan's work suggested growing the model much faster than the data, so a 10x compute increase meant scaling the model about 5.5x but tokens only 1.8x. Chinchilla found the opposite: model and data should grow together, doubling tokens every time you double parameters. The 70B-parameter Chinchilla beat the 280B Gopher under the same budget. Much of the gap came down to small choices, like whether you count embedding parameters and how large the test models were.

The part worth reading is about fragility. Weng shows that fitting a scaling law is sensitive to things that sound trivial: rounding the loss, averaging instead of summing it, or which range of runs you fit over. Her post revisits how a reanalysis of Chinchilla's own method shifted its conclusions once the rounding and loss handling were corrected. Data also stops being free once you exhaust unique tokens, and each repeat is worth less than the last.

Why it matters

If you plan training runs from small-scale fits, your predicted optimal model size can swing widely on bookkeeping choices most teams never write down. Before committing a large budget, check how your own fit moves when you change how you count parameters and sum the loss.