Why model routing is an optimization problem, not a classifier

An IBM Research team pushes back on a common assumption: that routing between AI models is just a classifier that sends easy tasks to a cheap model and hard ones to an expensive one. Their argument, backed by measurements, is that this framing hides most of what actually sets cost and speed.

The clearest example is pricing. On 417 tasks from the AppWorld benchmark, Claude Sonnet cost $79 in total, about $0.19 per task, while GPT-4.1 cost $155, roughly double, at $0.37 per task. That reverses what the published per-token prices would suggest. The reason is that agent workloads reuse large chunks of context across many steps, and Sonnet's cheaper cache reads more than made up for its higher base price and longer reasoning. Difficulty is just as slippery: a prompt that looks trivial can trigger hidden retrieval and compliance checks, while a scary-looking technical prompt might be handled well by a small specialized model. Latency, too, is often set by infrastructure like cache warmth and endpoint congestion rather than the model's raw speed.

So the team reframes routing as an optimization problem. They build a cost-accuracy frontier and pick a point on it: one configuration reached 84% accuracy for $93 and 83 seconds, a 21% cut in cost and 9% in latency against their baseline, at the price of 4% accuracy.

Why it matters

If you run a multi-model setup and chose the router from per-token price sheets, your bill may be higher than it needs to be. The practical move is to measure real per-task cost with caching on your own traffic, then decide how much accuracy you are willing to trade for a large cut in spend.