Why self-tuning research agents don't overfit

Machine learning theory has a warning that every student learns: if you keep testing ideas against the same validation set, you will eventually fit the noise rather than the signal. So a puzzle has been nagging at researchers who run large language models as automated research agents. These agents propose a strategy, score it on a held-out set, adjust, and repeat for hundreds of rounds, exactly the loop that should produce overfitting. Yet in practice they often keep generalizing. A team at Amazon Science set out to explain why.

Their experiment used three agents in sequence. An explorer optimized strategies against a validation set over many rounds. A compressor then squeezed each winning strategy into a very short prompt, between 16 and 32 tokens across eight datasets. Finally a reproducer, given only that compressed prompt and the training data but no access to the validation set, tried to match the original result. It usually did. When the researchers forced an agent to overfit on purpose, the trick broke: the gains vanished once the strategy had to pass through the narrow prompt.

The explanation they offer is an old one made concrete. A strategy that fits into 16 tokens cannot secretly memorize a dataset, so whatever survives the squeeze must describe real structure. "What fits doesn't overfit," as they put it, which turns compression into both an explanation and a practical test for whether an agent learned something real.

Why it matters

If you build agents that tune themselves against a benchmark, you can use compression as a cheap honesty check: force the winning recipe into a few tokens and rerun it fresh. What survives is probably a genuine method, and what falls apart was overfitting you would otherwise have shipped.