The retrieval question that actually matters this year isn't "which embedding model." It's whether you let an LLM drive the search loop. A new NVIDIA NeMo Retriever paper, *Beyond Semantic Similarity*, finally prices that decision out, and the numbers are blunt enough to design around.
The setup is unremarkable by design: a ReAct loop where the agent plans, issues `retrieve(query, top_k)` calls, fuses candidates, and ranks documents until it has enough context. Holding the embedding model fixed, agentic retrieval improves nDCG@10 by 8.7 points over standard dense retrieval, and the same pipeline stays competitive on both the ViDoRe v3 and BRIGHT leaderboards — which is the more interesting half, because specialized retrieval methods tend to fall apart out of domain. The cost: 107.4 seconds per query against 0.67 for standard retrieval, and 764.1K input plus 5.8K output tokens per query .
That's roughly 160x the latency and four to five orders of magnitude more token spend per query. Nobody is putting that behind a search box.
What it changes in your architecture
The honest read is that agentic retrieval is a *tier*, not a replacement. You keep one-pass hybrid search as the default path and route to the agentic path on queries where a wrong answer is expensive — compliance lookups, multi-hop research over a document corpus, anything where a human would otherwise spend twenty minutes. The 107-second budget is fine when the alternative is a person; it is absurd when the alternative is a dropdown.
Worth noting that this is shipping, not just a paper. NeMo Retriever exposes the agentic path through `retriever query --agentic`, `POST /v1/query` with `agentic=true`, and an `agentic_query` MCP tool, running the ReAct loop over the same LanceDB table that one-pass retrieval uses. It also ranks documents rather than chunks while returning the same hit fields — a quiet but real change if your downstream prompt assembly assumes chunk-level granularity. Operationally, a self-hosted vLLM-backed NIM has to enable automatic tool choice and a tool-call parser, and the Helm `answer_llm` config doesn't turn those on by default . That's the kind of footgun that costs an afternoon.
The cheaper counterargument
Before you commit to embedding-driven agent loops, there's a competing line of work worth reading. One approach drops similarity search entirely in favor of grep, shell, and read tools over the raw corpus: swapping Qwen3-Embedding-8B for direct corpus interaction on the same Claude Sonnet 4.6 backbone raised BrowseComp-Plus accuracy from 69.0% to 80.0% while cutting cost from $1,440 to $1,016 . Better *and* cheaper, on that benchmark.
Which suggests the expensive part of agentic retrieval isn't the reasoning — it's shoveling embedding hits through the context window. NVIDIA's stated next step is distilling these loops into smaller specialized open-weight agents . Until that lands, treat the agentic path as a premium tier with a hard query budget, and instrument what fraction of queries actually justify it.