Percona publishes vector index benchmark and explains why almost every vector search number is empty

Practically every database today advertises vector search, and almost all of them publish a post with a large queries-per-second number. The problem, according to Evgeniy Patlan in an article on the Percona Database Blog, is that these numbers can rarely be verified, because the information that would make them meaningful is usually missing. To tackle this, Percona built a tool called vector-bench: the operator names the engines, each one is assembled from pinned versions, runs in the same container, on the same cores, with the same data and the same measurements, and generates a report.

For the Brazilian DBA who's starting to see demand from AI and RAG applications, the core message is sober and worth repeating: in vector search, every measurement is a pair. Throughput without recall next to it says nothing, and recall without throughput is free.

What's being indexed

An embedding is a fixed-size array of floats that comes out of a model. The useful property is that semantically similar inputs end up close together when you measure the distance between them. Two metrics cover almost everything: L2, the classic Euclidean distance extended to many dimensions, and cosine similarity, which measures the angle between two vectors and ignores their length. Which one applies is a decision made by the model that generated the embeddings, not a query-time choice. Swapping one for the other produces a meaningless result.

The desired query is simple to state: "the 10 rows whose vectors are closest to this one."

SELECT id FROM documents ORDER BY distance(embedding, ?) LIMIT 10;

The problem is answering this exactly. Calculating the distance from the query vector to every row and sorting is, in practice, a full table scan with a lot of arithmetic on top. No B-tree or hash helps here: they don't know how to sort a million points by proximity in 1,536 dimensions. A vector index trades away exactness to avoid this, looking at a few thousand promising candidates instead of every row. This is approximate nearest neighbor search, or ANN, and it gets it right "almost always." Measuring that "almost always" is the benchmark's job, and it depends on knowing the ground truth: the true neighbors for each query, calculated once by brute force, without an index. This is exactly the number that most vector search claims leave out.

HNSW and IVF: two opposite designs

Almost every database has chosen one of two designs, and that choice defines what can be tuned.

HNSW (Hierarchical Navigable Small World) is a layered graph of vectors, a close relative of a skip list. The top layer has few nodes with long links; each layer below has more nodes and shorter links. The search starts at the top, always jumps to the neighbor closest to the query, and drops a layer when there's nothing closer. Two parameters matter: M, how many links each node keeps (fixed at build time; higher means better recall at the cost of slower builds and a larger index), and ef_search, how many candidates the search tracks as it walks. The latter is a session variable, adjustable per query. There's also ef_construction, the same idea applied during construction, which not every engine exposes.

IVF (Inverted File) partitions instead of linking. At build time, it groups vectors into nlist clusters with a representative at the center; at query time, it compares against the representatives, picks the nprobe closest clusters, and searches only inside them. It builds much faster and uses less memory, but it usually gives worse recall at the same speed, and it fails when the true neighbor sits just outside the clusters visited. Percona only tests engines running HNSW, which is what most databases have shipped; IVF-only engines are set aside in a separate bucket, so the benchmark doesn't end up measuring the distance between two algorithms instead of implementation quality.

Why one number is never enough

Recall isn't a property of the engine, it's a setting, and ef_search is the knob. The article shows the same HNSW index, on the same machine, with the same data, varying only how many candidates the search tracks:

ef_search=10    3,678 queries/sec   recall 0.9593
ef_search=800     409 queries/sec   recall 0.9987

Keeping 800 candidates instead of 10 finds a better answer and takes nine times longer. Both lines are honest measurements of the same index. That's why "our database does 3,678 vector queries per second" tells you nothing: you don't know how often it was returning wrong rows, and whoever quotes the number might not know either. The reverse is just as empty, because recall 1.0 is always available: just turn off the index and scan the table.

The pitfalls that inflate any number

The benchmark creates one table per engine, with an id, an integer tag column used only in the filtered tests, the vector, and an HNSW index with M configured. The tag values run from 0 to 99, uniformly distributed, so tag < 10 passes about 10% of the rows and tag < 1 about 1%, controlling selectivity. And every engine has at least one setup detail capable of quietly wrecking the numbers:

• TOAST in PostgreSQL. A 1,536-dimension vector counts as a large value and gets stored out of line. Without setting the column to STORAGE PLAIN, every distance comparison pays an extra fetch. It's one line of DDL; forgetting it makes PostgreSQL look slow for a reason that has nothing to do with your vector search.
• Incremental vs. bulk build. In incremental mode the graph is updated on every INSERT and the table is ready to query right away; in bulk mode, everything loads first and the graph is built in one pass, much faster, but the table doesn't answer queries until it's done. One engine in the set does both, and its bulk path loaded 18 times more rows per second than its own incremental path. Comparing one engine's bulk mode against another's incremental mode compares two ways of building an index, not two engines.
• Silent full scan. Any of these engines can stop using the index and scan the table. Since the scan returns an exact but slow result, it shows up as high recall and low throughput, indistinguishable from a conservative index, unless you read the execution plan. One optimizer switches to the scan when the LIMIT exceeds about a quarter of the table; another falls back silently, without error or warning, when the query asks for a different distance than the one the index was built for (build for cosine, query with the L2 operator, and out comes a Seq Scan with sort, with nothing warning you). That's why each driver runs EXPLAIN and checks whether the index name shows up in the plan, emitting WARNING: vector index NOT used when it doesn't.

Filtered search, the case that justifies keeping vectors in the database

Filtered search (WHERE tag < ? ORDER BY distance(v, ?) LIMIT 10) is the scenario that supposedly justifies keeping vectors in the relational database instead of in a dedicated store, and it deserves attention for that reason. Filtering changes what "correct" means: the true top 10 among rows with tag < 10 isn't the overall top 10, so the ground truth has to be recalculated by brute force only over the rows that pass, for each selectivity level. Scoring against the ground truth of the full dataset gives near-zero recall on every engine, and Percona admits having made exactly that mistake for a while.

There's a revealing detail about the order of operations: HNSW searches by distance first and only applies the WHERE afterward. It gathers a few thousand candidates, the filter discards most of them, and sometimes fewer than 10 rows are left. In one run, 81 out of 200 queries came back incomplete, even with about 99,000 rows passing the filter at 10% selectivity. "10 rows, four wrong" and "six rows, all correct" both score 0.6, but the first case needs a wider search and the second needs iterative scanning; that's why the report gives the count separately.

How the comparison is kept fair

Everything runs twice. The normalized pass gives every engine identical CPU, memory, and cache, so that any difference in the result belongs to the implementation. The tuned pass lets each engine use what its own documentation recommends. A result that survives both passes is about the engine; one that flips between them is interesting for a different reason. Cores are pinned one per physical core (SMT siblings share execution units), never mixing P-cores and E-cores on hybrid chips, and durability is relaxed equally across all of them, otherwise the benchmark would be comparing fsync policies under the label of vector search.

One hardware note catches people off guard: several implementations ship hand-written AVX-512 code for the distance math, where a single instruction handles the arithmetic for 16 floats at once. The same index on a CPU without AVX-512 is practically a different benchmark, and the slowdown isn't the same for every engine, so you can't even scale the numbers to compensate. CPU model, feature flags, engine versions and commits, image IDs, and effectively resolved resource limits all go into the manifest for each run. No manifest, no report.

The post's honesty extends to its own mistakes: the first ingest numbers measured 88 rows per second because the client was doing one INSERT per round trip with autocommit on; batching 500 rows per transaction took the same engine to 373. And the two passes even ended up sharing a results directory at one point, causing the tuned pass to skip everything the normalized pass had already calculated, so the "tuned" numbers were almost all normalized numbers under a different label. For anyone evaluating a vector database in Brazil, the practical lesson is direct: before believing any number, ask for the ground truth, the recall/throughput pair, and the silent-scan check. If the benchmark doesn't mention that, there's reason to be suspicious.