Epoch tested AI on its own jobs, and it couldn't do them

Epoch AI ran an unusual experiment: it gave six frontier models 11 real tasks from its own work, spanning graphic design, Data Insight and Data Explorer generation, data center research, and research design. Each model ran at its highest reasoning setting with the tools and references a new hire would get, and a staff member graded the output against rubrics built with the team. The conclusion is that models handle well-defined work but cannot yet produce Epoch-quality output on their own.

The failures are specific. Models miss implicit conventions like visual style and what the audience cares about, and they tend to write denser than Epoch does. On research they suggest promising directions but design vague experiments, and they sometimes present artifacts of their own setup as findings. GPT-6 Astra used a 4096-token budget that cut off 61 of 280 responses, then framed the truncation as "sensitivity to the acquisition budget". On open-ended tasks the models often converged, with three of six picking the same polling topic. Kimi K3, which scores 158 on Epoch's Capabilities Index, filtered all arXiv papers instead of physics papers and reported a wrong chart. Claude Fable 5.1 and GPT-6 Astra tied at the top, while open-weight models trailed even on well-defined tasks.

Progress is real at the edges. A task that failed earlier in 2026, porting an article to Substack through computer use, now succeeds, and Epoch dropped a graphic-restyling task from the suite because Fable 5.1 handled it well.

Why it matters

Benchmarks reward easily verifiable tasks and can make models look readier for real jobs than they are. If you are deciding whether to hand a workflow to an agent, test it on your actual work and watch for confident results that come from the agent's own mistakes.