An AI agent reviewed his release and found a silent data-loss bug

Simon Willison spent about $149 having an AI agent review a release of sqlite-utils before he shipped it, and the review paid for itself on the first serious bug. He pointed Claude Fable at the 4.0rc2 codebase and worked through 37 prompts and 34 commits, changing 30 files and roughly 1,300 lines. The most severe find was a quiet one: delete_where() never committed its transaction and left the database connection in a poisoned state. Rows deleted, rows inserted afterward, and new tables all silently vanished on the next reconnect. That is the kind of bug that ships in a release candidate and corrupts real users' data before anyone notices.

The cost breakdown is worth seeing because it is unusually transparent. The main session ran $141, and a handful of scoped sub-agents that swept the API surface, transactions, and migrations added a few dollars each. Willison then ran a second pass with GPT-5.5, which caught two more commit-handling bugs in db.query(), including one where INSERT ... RETURNING only committed after the caller fully drained the iterator.

His takeaway is that having one model review another's work "really does work," and he now does it by habit. He also admits he should have pushed more of the routine sweeps onto cheaper sub-agents to cut the bill. The full writeup includes the per-agent costs.

Why it matters

If you ship libraries with an AI coding agent, a second-model review pass before release is cheap insurance against exactly the bugs your tests miss. Budget for it, and use small sub-agents for the mechanical sweeps so the cost stays in the tens of dollars, not hundreds.