Is grep enough? A new paper says yes, with caveats

A new paper by Sahil Sen and colleagues, posted to arxiv as "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search," runs a head-to-head between grep-style text search and embedding-based retrieval inside real LLM agent harnesses. The result is the kind of finding that should annoy people who have shipped a vector database: grep consistently beats vectors on accuracy across the tested setups.

The experiments are concrete. The authors evaluate 116 questions from LongMemEval across four harnesses, Chronos, Claude Code, Codex, and Gemini CLI, comparing how tool outputs are presented inline versus through files. A second experiment progressively adds unrelated conversation history to measure robustness as the haystack gets noisier. Grep holds up better. But the bigger headline is that performance varied more between harnesses than between retrieval methods, even on the same data. The model and the way it sees tool output mattered more than the search algorithm doing the lookup.

The paper does not claim vectors are useless, and the tasks here are long-memory recall, not semantic association at scale. The point is narrower: when an agent can run a fast text search itself, the embedding pipeline you spent two months tuning may not be earning its keep.

Why it matters

If you build agents, the cheapest thing to try this week is replacing a vector lookup with a tool the model can call grep through, and measuring whether quality drops. If you build harnesses, the result is more pointed: how you present tool output likely matters more than which retrieval engine you bolted on.