BigDATAwire reported on September 3 that Qdrant is arguing AI search has a benchmarking problem, and wants to address it at 10-billion-vector scale. The headline number is the least interesting part. The useful part is the admission underneath it: almost every ANN benchmark the field cites is run in a regime that does not resemble a production knowledge base.
Where the published numbers stop transferring
At a million vectors, everything works. HNSW fits in RAM, recall@10 sits above 0.95, p99 stays in single-digit milliseconds, and the choice of engine barely matters. The physics change somewhere past a billion. The graph no longer fits on one node's memory, so you are into memory-mapped or on-disk indexes, and into quantization — scalar, product, or binary — where you trade recall for a 4–32x footprint reduction and then buy some of it back with a rescoring pass over full-precision vectors. Every one of those decisions has a different cost curve at 10B than at 10M, and none of it is captured by a static, unfiltered, single-tenant benchmark run on a uniformly distributed academic dataset.
Three things production retrieval does that standard benchmarks don't measure:
*Filtered search.* If your corpus is an enterprise knowledge base, nearly every query carries a permission filter. ACL-constrained search degrades HNSW's graph connectivity — the neighbors you need may all be filtered out, and the traversal wanders. Recall under a 0.1%-selectivity filter is a genuinely different number from unfiltered recall, and it's the one you actually ship against.
*The write path.* Benchmarks build the index once. Real systems take continuous upserts, deletes as tombstones, and background segment merges. Recall and latency after a week of ingestion are not the recall and latency you measured on day zero.
*Multi-tenancy.* Thousands of tenants sharing nodes changes the memory story completely, and single-collection throughput tells you nothing about it.
What to do instead
Build a recall harness on your own corpus. Take a sample, compute exhaustive brute-force ground truth for a few thousand real queries, then measure recall at your actual filter selectivity, with your actual quantization config, after a realistic period of writes. It's a day of work and it will disagree with the vendor charts.
The other move this month
AWS added a native ServiceNow connector to Bedrock Managed Knowledge Base in September, plus a user-managed setup path for SharePoint, OneDrive, and Confluence. The managed KB went GA in June. Read together with MCP's enterprise-managed authorization work — the protocol blog covered it in July, and WorkOS wrote about GA and a new grant type two weeks ago — the competitive surface is clearly shifting from index quality to connector coverage and identity propagation.
Who should care about the benchmarking thread: if you're under 50M vectors, you aren't losing recall to ANN parameters. You're losing it to chunking and reranking. Fix those first.