Using DSPy to find a hidden bug in an agent's prompt

Simon Willison spent a day using DSPy, a framework for optimizing prompts against measurable metrics, to improve the system prompt behind Datasette Agent, the tool that answers questions by writing and running SQL. Instead of tuning the prompt by hand, he handed most of the work to Claude Code: install Datasette, datasette-agent, and DSPy, then build a harness that runs DSPy's optimizer against the agent's real tools. The harness pointed small, cheap models (GPT 4.1 mini and nano) at a live in-process Datasette instance, generated a gold-standard set of questions and answers automatically, and scored the agent with custom metrics. You can read the full write-up for the setup.

The useful result was not a magic prompt but a specific bug the process exposed. The agent's schema listing showed only table names, and a line in the prompt told the model not to call describe_table if it already had what it needed. Those two choices combined to push the model into guessing column names, which then failed and triggered slow error-and-retry loops. The fix is dull and concrete: put column names in the schema listing, or drop the advice that discourages checking. Willison's takeaway is that DSPy earned its keep here as a way to measure the effect of a prompt change, not as an automatic prompt writer.

Why it matters

If you build SQL or tool-using agents, the lesson is that a thin schema plus a "don't look it up" instruction can quietly cause retry loops that burn tokens and latency. A measurement harness makes those failures visible before your users hit them, so it is worth building one before you start hand-editing prompts.