Turning a database benchmark into reinforcement learning rewards

Microsoft's ThinkingBox benchmark scores an AI agent by inspecting the database it leaves behind rather than reading its final message. In a new write-up, the team shows that the same executable checks can serve as reward signals for reinforcement learning. Each training episode gets a 1 if the final database state is correct and a 0 if it is not, with a small share of tasks using an LLM judge for details the database cannot capture.

The careful part is choosing which tasks to train on. A task is only hard or easy relative to the current model, so the team ran the base model against each candidate many times, estimated its success rate, and kept only tasks that landed between 8% and 90%. Tasks the model always fails or always passes give the GRPO algorithm no contrast to learn from, so they were dropped, leaving a fixed pool of 187. Training used full-parameter GRPO on a 27-billion-parameter Qwen model for 50 steps across 24 H100 GPUs.

The results favor consistency over coverage. Average pass@1 rose from 51.7% to 60.8%, and the number of tasks the model solved on all 20 attempts jumped from 38 to 107. Tasks solved at least once actually dipped slightly, from 453 to 446, so the model became more repeatable rather than more capable. An internal cost index for each dependable task fell from $24.36 to $8.00.

Why it matters

If you post-train agents, this is a recipe you can copy: deterministic end-state checks double as rewards, and probing task difficulty before training tells you whether your data can teach the model anything at all.