An API change that helps big models can break small ones

When you add a feature to make a library easier for coding agents to use, you probably test it with a strong model and assume the gain carries down. Hugging Face's agent benchmarking write-up shows that assumption is dangerous. The team built a harness that measures not just whether an agent gets the right answer but how much it spends getting there: tokens used, time taken, errors made, and which APIs it actually calls. Each task runs across three setups, from a plain install, to the full source repository, to a curated set of docs and examples placed in context.

Testing changes to the Transformers library produced a clear split. For large models like Kimi, GLM-5.1 and MiniMax-M2.7, a new command-line interface plus documentation cut median execution time, though it pushed input tokens up by roughly 60 percent, from about 4,000 to 6,400 on the source-repo setup. Adoption of the new CLI only reached 55 percent once the docs were in context, which suggests good documentation is what drives feature discovery. Smaller models told a different story. Qwen3-14B's sentiment-classification accuracy fell from 100 percent to zero once the CLI docs were present, because it treated the documentation as a tool it could call, and its token use jumped tenfold without any benefit.

Why it matters

If you maintain a library that agents consume, an affordance that clearly helps frontier models can quietly poison results for the 4B to 14B models many teams run for cost reasons. Test agent-facing changes across model sizes before shipping, rather than trusting that what helps the strong model is harmless for the weak one.