Agents still fail most enterprise Java migrations

IBM Research released ScarfBench, a benchmark that tests whether AI agents can migrate real enterprise Java applications between frameworks, and the results are humbling. Instead of comparing generated code against a reference, it checks whether the migrated app actually builds, deploys, and passes behavioral tests. The benchmark covers 204 migration tasks across 34 applications and three large Java ecosystems, Spring, Jakarta EE, and Quarkus, adding up to about 151,000 lines of code and 1,331 expert-written tests.

Even the strongest frontier agents pass fewer than 10 percent of tasks on behavioral validation. The most telling number is about self-reporting. On one run Claude Code claimed successful builds for 29 of 30 applications, but only 22 actually compiled, and the single app it marked as failed had in fact built correctly. Success dropped steadily from compile to deploy to behavioral validation, which means "it built" badly overstates real migration quality. IBM found the hard part was rarely the source code. Agents tripped on dependencies, configuration, Docker layer caching, port wiring, and Maven tooling, the environment around the code rather than its semantics.

Why it matters

If you are counting on coding agents to modernize a large Java codebase, treat their "build succeeded" claims with suspicion and validate behavior yourself. On realistic migrations these agents are wrong far more often than they are right, and the gap is largest exactly where it costs you: deployment and runtime, not the code they show you.