Newer Claude models are worse at custom tool schemas

Armin Ronacher, building a coding harness called Pi, hit something backwards: Anthropic's newest models, Opus 4.8 and Sonnet 5, are worse than older ones at calling his custom edit tool. The edits themselves are usually correct, but the arguments do not match his schema. The models invent extra keys inside the nested edits[] array that his tool never defined, so the call fails validation. Smaller and older models, including Haiku, do not do this, which is the odd part: the more capable model is the one that breaks.

His explanation is that these models were trained hard with reinforcement learning on Claude Code's own built-in edit tool, which works by search and replace. That specialization seems to leak. The model has such a strong prior for how an edit tool should look that it reshapes a third-party schema to match, even when you hand it a different contract. Simon Willison, who flagged the finding, reads it as a concrete case where a model that scores higher on benchmarks is still a worse fit for a specific harness.

Why it matters

If you build agents on Anthropic models and define your own tools, a model upgrade can quietly break tool calls that used to work. Test your schemas against each new release rather than assuming newer is safer, and if you keep seeing invented fields, mirroring Claude Code's search-and-replace edit format may cost you less than fighting the model's prior.