Most teams shipping a memory layer evaluate it the same way: run a multi-session QA benchmark, measure answer accuracy, ship. A paper posted to arXiv on October 15 ([2610.22209](https://arxiv.org/abs/2610.22209), from authors at AWS and the Technical University of Cluj-Napoca) argues that this measures almost everything except memory.
The failure is a conflation. Downstream QA accuracy is the product of three systems — write-path memory construction, retrieval, and generation — and scores only the composite. That produces errors in both directions. False positives come from LLM compensation: a strong generation model reasons over partial retrieval results and produces the right answer despite a corrupted or incomplete memory state, which extrinsic evaluation scores as success. False negatives come from the retrieval bottleneck: memory stores and updates the fact correctly, but embedding mismatch or a top-k cutoff means it never reaches the context window, and the eval blames memory.
What FEB-30 actually does
The benchmark, FEB-30, is built by inverting the usual construction order. Instead of annotating documents for facts, it defines 30 fact-evolution trajectories first — across 6 diagnostic categories and 3 difficulty levels — then generates the multi-author artifacts (documents, emails, conversations) that express them. Ground truth is deterministic by construction, with no human annotation, and because you know what the store *should* contain at every step, you can inspect it after each ingestion rather than probing it through a question.
The gap this opens up is large. On a representative memory system, intrinsic (store-level) and extrinsic (QA) scores diverged by 16 percentage points on average, with 44% of trajectories past a meaningful divergence threshold. The sharpest finding: the system classified memory operations — create, update, reinforce, supersede — correctly only 20% of the time. Downstream QA detected none of this. Benchmark, generation pipeline, and scoring framework are released at [amazon-science/feb-30-benchmark](https://github.com/amazon-science/feb-30-benchmark).
Why it matters if you run this in production
The operation-classification number is the one to sit with. Supersede-versus-reinforce is exactly the decision that determines whether your agent still believes a policy that changed in March. QA accuracy stays flat while that rot accumulates, because the generator papers over it until the day it can't.
This lands alongside a related result from the same week: a 654-issue field study of RAG configuration bugs across 10 open-source systems ([2610.18567](https://arxiv.org/abs/2610.18567)) found that in roughly a third of cases, the stage where a symptom appears is not the stage where the root cause lives — and when the two are far apart, resolution time rises by up to 2.83×.
Same lesson from two directions: end-to-end metrics on multi-stage pipelines tell you something broke, not where. If you operate a memory layer, add a store-level assertion to your eval harness this quarter. LangChain's Deep Agents post from October 14 makes the complementary architectural argument — offload context to inspectable files rather than truncating it, since recoverable beats lost. Inspectable also beats opaque when you're trying to find out what your agent actually believes.