ThinkingBox grades AI agents on the database, not the transcript

Microsoft and Hugging Face built a benchmark called ThinkingBox around a simple observation: an agent's trajectory is a claim, and the database state is the evidence. Most agent evaluations read the final reply or the sequence of tool calls and decide from there whether a run succeeded. ThinkingBox ignores the talk and inspects the backend state the agent actually left behind.

The motivating example is a customer whose delayed appliance order was stuck in a carrier exception. The agent made nine well-formed tool calls, correctly worked out that she was not eligible for compensation, and closed the ticket as resolved. But the required end state was "on hold", because the carrier exception was still open. The transcript looked clean while the database showed the error.

The scale makes the gap hard to dismiss. ThinkingBox runs 507 stateful business workflows, each 20 times from an identical clean backend, and compares the recorded side effects against a required end state. Across 121,680 trials over 12 models, nearly 80,000 attempts failed the state checks, and about two thirds of those failures ended cleanly and even invoked a state-changing tool without reporting an error. Claude Opus 5.5 leads pass@1 at 67.16%. Kimi-K3, the strongest open-weight model, solves 93.89% of tasks at least once but only 13.41% on all 20 attempts. About 80% of failures were tool-usage problems rather than reasoning errors: failing to recover from a tool error, a failed precondition, an empty lookup.

Why it matters

If you deploy an agent that writes to real records, a single-run score and a confident closing message will overstate how reliable it is. Measure repeated runs and check the end state, and plan for the distance between an agent that passes once and one that passes every time.