Nathan Lambert on what a leading AI benchmark can and cannot tell you

Artificial Analysis publishes the most-cited "intelligence index" for frontier models, and its new v5 rebuilds the scoring into four equally weighted buckets: knowledge work, coding, science, and general ability, 25% each. Nathan Lambert uses the release to lay out what he calls the benchmarker's dilemma. An index has to collapse many results into one number so people can compare models, but every weighting choice encodes a worldview, and the wiggle room lets an evaluator nudge the result toward a preferred story. Trust is the harder problem. Artificial Analysis earns money from the same labs it ranks, and Arena runs pre-release tests for labs. Lambert says he trusts non-profits and government bodies most for fairness and startups most for accuracy at the frontier.

On the substance, he reads v5 as reasonable but strained. GPT-6.1 Terra ranks above Claude Fable 5.1, which he takes as the index tuning toward a general agentic collaborator over narrower skills. Eight closed models beat the best open-weight model, Kimi K3.1 Pro, and the widest gaps show up on science evals like Terminal-Bench-Science and CritPt v2. His argument is that open models were never built to win at the very frontier; their value is pushing inference toward commodity pricing and letting companies specialize intelligence on their own data. He is uneasy about one v5 change: hidden prompts answer the benchmaxxing problem but remove the community's ability to find bugs, and they require full trust in the builder. His fix is more indices with different biases, including domain-specific ones. The full piece works through each category.

Why it matters

If you pick models by leaderboard, this is a guide to reading one critically: check who funds the evaluator, what the weights favor, and whether the prompts are public. A single ranking is a starting point, not a verdict.