Your agent passed the test. IBM asks whether it will pass five times

A ReAct agent running on GPT-4.1 succeeds about 77.4% of the time on average, which sounds solid until you ask a harder question: how often does it succeed on all five runs of the same task? Only 53.0% of the time. IBM Research calls this the consistency gap, and here it is 24.4 points wide. The cause is subtle. When the model assigns similar probabilities to several possible next tokens, tiny platform differences like floating-point rounding or batching can flip which one it picks, and those flips compound across dozens of sequential decisions in an agent loop.

Their fix does not require a bigger model. A tool called the Consistency Analyzer takes one trajectory and replays each decision point with five resampled completions, without rerunning the whole task, to find the steps most likely to flip. Those steps become plain-language guidelines, for example "use regex for checkbox counting rather than substring matching," which feed back into the agent through IBM's existing ALTK-Evolve system. The results are concrete: Pass^5 rises from 53.0% to 69.0%, average accuracy holds at 81.0%, and the guidelines transfer to similar tasks for a 13-point gain. The team's advice is to report Pass^k next to Mean@k, because an average hides how often an agent quietly fails on a repeat.

Why it matters

If you run agents in production for anything repeatable, like reconciliation or data extraction, average success rate is the wrong number to trust. Measure how often the same input succeeds every time, and diagnose the unstable steps before you reach for a larger, more expensive model, because consistency and raw capability are separate problems.