Turbopuffer rebuilds its engine and demotes vector search to a side index

Turbopuffer, a search engine used for retrieval behind AI applications, has published a frank engineering post on why its original design was holding it back. The title, "RIP, vector database," is pointed: the company is moving away from making the vector index the primary key around which everything else is stored, and treating approximate nearest-neighbor (ANN) search as just another secondary index instead.

The reasons are concrete. When documents carry several vectors each, a vector-primary layout duplicates all the non-vector fields once per vector, which inflates storage. Writes are worse: rebalancing vectors can cascade through a document's other attributes, so a single update ends up moving hundreds of fields. The old structure also fought the query engine, because different engines want very different block sizes (DuckDB works in 2,048-row chunks, ClickHouse near 65,000, ANN clusters in the low hundreds) and all of them were pinned to the ANN key. Turbopuffer points to its earlier full-text search rework as proof the new approach pays off: switching from cluster-boundary postings to fixed 256-document blocks made those indexes 10 times smaller and up to 20 times faster.

This is not a small system. The company says it now holds more than a trillion documents, takes over 10 million writes per second, and serves more than 25,000 queries per second, with single indexes of 100 billion vectors answering 200 millisecond reads at the 99th percentile.

Why it matters

If you run retrieval for an AI product, this is a clear argument that a pure vector database can become the bottleneck as your data grows past one vector per row, and a signal to weigh general-purpose search that handles filters, text, and vectors in one engine.