An open leaderboard for whole agent systems, not just models

Most agent benchmarks score a model. IBM Research argues that misses where production cost and quality actually come from, so its new Open Agent Leaderboard evaluates the whole system: the model plus the agent scaffold around it. It runs each agent through one unified protocol across six benchmarks, including SWE-Bench Verified for bug fixes, BrowseComp+ for web research, AppWorld for multi-app personal tasks, and three tau2-Bench customer-service suites. Agents are tested as general-purpose systems with no benchmark-specific tuning, and every run reports both success rate and dollar cost.

A few results are worth acting on. The same model wrapped in different agents lands in very different places, so the scaffold is not a detail. Failed runs cost 20 to 54 percent more than successful ones, which means an agent that fails slowly is expensive in production, not just annoying. Tool shortlisting, giving the agent a smaller relevant tool set, improved performance for every model tested. General-purpose agents often matched systems hand-built for a specific benchmark. The two open-weight models added so far, DeepSeek V3.2 and Kimi K2.5, trail the closed frontier by 18 to 29 percentage points on average. The leaderboard, the paper, and the Exgentic framework behind it are all public.

Why it matters

If you are picking an agent stack, this gives you a cost-aware comparison instead of a single accuracy number, plus two cheap wins to try first: shortlist the tools you expose, and measure what your failures cost before they reach production.