How frontier post-training got complicated, and why distillation now sits at the center

Nathan Lambert and Finbarr Timbers trace how the recipe for turning a base model into a useful assistant has changed over four years, and the short version is that it keeps getting more complex. The 2022 InstructGPT approach was three clean stages: supervised fine-tuning, a reward model, then reinforcement learning. By 2025, DeepSeek R1 had pushed large-scale reinforcement learning to the center of the process rather than a final polish. In 2026 the recipe fragments further, with labs training several domain specialists and then merging them back into one model.

The pattern they single out is multi-teacher on-policy distillation. Running math, code, and agentic reinforcement learning together in a single job gets expensive, and the objectives start trading off against each other. So teams instead train separate specialist teachers, then let a general student model generate its own responses and learn token by token from whichever teacher is relevant. There is a real catch they flag: NVIDIA found that teachers trained with very different pipelines cannot simply be distilled together, so the merge step is harder than it sounds. Lambert also notes that DPO, a preference-tuning method that was standard two years ago, has largely dropped out of frontier recipes, though he is honest that it is unclear whether it stopped working or labs just moved on.

Why it matters

If you work on post-training, the default is no longer one big reinforcement learning run. Plan instead to build and combine specialist teachers through on-policy distillation, and budget time for the distribution-mismatch problems that show up when you try to merge models trained different ways.