Coding agents lose 30 points when you tighten backend constraints

Dente, Satriani, and Papotti put together one of the more honest evaluations of coding agents we have seen recently. Their paper, posted to arxiv this week, measures how agent performance changes as you tighten the requirements on a backend task. They ran 80 greenfield generation jobs and 20 feature-implementation jobs across 8 web frameworks, using a fixed unified API contract and two kinds of checks: end-to-end behavioural tests and static code verifiers.

The headline number is blunt. Capable agent configurations lose about 30 percentage points in assertion pass rate when moving from a loose baseline to a fully specified task. They call this constraint decay: as structural requirements pile up, performance drops sharply instead of gracefully. Framework matters too. Agents do reasonably well in minimal Flask environments and substantially worse in convention-heavy ones like FastAPI and Django, where the right answer depends on knowing the framework's idioms. The leading source of failure was the data layer: agents wrote queries that compose wrong or violated ORM runtime contracts. The full paper is at arxiv.org/abs/2605.06445.

This is consistent with what production teams have been saying for months: agents are great at the demo, then brittle when you put them inside a real codebase that imposes opinions. The paper at least puts a defensible number on the gap, broken down by framework and failure type.

Why it matters

If you are picking a coding agent or sizing how much human review it needs, evaluate it on your real framework with your real constraints, not on a clean greenfield benchmark. The 30-point gap is a useful baseline for how much you should distrust the demo.