Memindex

August 29, 2026

Recall benchmarks stopped predicting agent memory quality

LoCoMo and LongMemEval are saturating in the 90s while memory-in-the-loop benchmarks show the real failures — here's what to measure instead.

The agent memory numbers published over the past week are high enough to be uninformative. Mem0's benchmark breakdown, posted about five days ago, reports LoCoMo at 92.5% and LongMemEval at 94.4% — and BEAM 1M at 62%. EverMind's August update claims state-of-the-art across four long-term memory evaluations, including 90.04% recall on HaluMem, alongside LoCoMo, LongMemEval and PersonaMem v2.

Two benchmarks sitting in the low-to-mid 90s across competing systems have stopped discriminating between them. The 30-point drop on BEAM is the number that still carries information.

The gains are on the write path, not the retriever

The more useful detail in Mem0's state-of-the-field post is where its April 2026 algorithm actually improved. Built on single-pass hierarchical extraction and multi-signal retrieval, its two largest gains over the prior algorithm were temporal queries (+29.6 points) and multi-hop reasoning (+23.1). Those are the categories where facts accumulate, change, and reference each other — exactly the cases flat top-k retrieval handles badly.

The design change behind that is a write-path decision: agent-generated facts are now stored as first-class, so the model's own confirmations and recommendations carry the same weight as user-stated facts. If your memory layer only ingests user turns, half the conversational state never gets written. No amount of retrieval tuning recovers it.

Mem0 also reports roughly 6,956 tokens per retrieval call on LoCoMo, and flags explicitly that this isn't directly comparable to the ~26,000 tokens per conversation in its 2025 paper — different units. Worth noting because per-call token cost is the metric that governs whether a memory layer survives contact with production traffic, and almost nobody reports it next to accuracy.

The replacement benchmarks measure memory as an action

Three efforts converge on the same critique. Microsoft's STATE-Bench is open source and memory-agnostic, with 450 tasks across customer support, travel and shopping covering policy compliance, information synthesis and multi-step reasoning; it asks whether agents improve with experience rather than whether they can recall a fact. MemoryArena frames the problem as multi-session Memory-Agent-Environment loops, arguing that existing evaluations assess memorization and action in isolation when the two are tightly coupled in practice. AMA-Bench makes a parallel point from the other direction: current benchmarks are dialogue-centric, while real agent memory is a continuous stream of machine-generated agent-environment interactions. LoCoMo-Plus and LongMemEval-V2 extend the originals, though Mem0 notes neither has displaced them.

A recent survey (arXiv 2603.07670) formalizes the shape this implies — memory as a write–manage–read loop coupled to perception and action — and points to work treating store, retrieve, update, summarize and discard as callable tools optimized end-to-end with RL.

Who this matters to

If you are choosing a memory layer this quarter, a LoCoMo number tells you almost nothing now. Ask for BEAM, a memory-agnostic harness like STATE-Bench, and tokens per retrieval call. If you are building one, the leverage is in what you decide to write and when you invalidate it — not in your reranker.

Sources

  1. [1] [2602.02007] Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation
  2. [2] A-MEM: Agentic Memory for LLM Agents
  3. [3] Knowledge and Memory Beyond RAG: Why 2026 Agents Need a Write Path, Not Just a Retriever | by Micheal Lanham | Apr, 2026 | Medium
  4. [4] AMA: Adaptive Memory via Multi-Agent Collaboration
  5. [5] AI Memory System vs RAG: Differences, Tradeoffs, and Use Cases
  6. [6] On the Structural Memory of LLM Agents
  7. [7] MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation
  8. [8] [2606.00610] MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation
  9. [9] Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers
  10. [10] 10 Best AI Tools for Knowledge Management in Large Enterprises (2026) | TTMS
  11. [11] What Is Enterprise AI Knowledge Management? 2026 Guide, FAQ & Trends - GoSearch FAQs + Answers
  12. [12] Enterprise Knowledge Base Platforms: Top 10 Picks for 2026
  13. [13] Guide to Enterprise Intelligence Systems 2026 | Bloomfire
  14. [14] Enterprise Search in 2025: How GoSearch Redefined AI-Powered Work | The GoSearch Blog
  15. [15] 2026’s Leading Enterprise Knowledge Management Systems
  16. [16] The definitive guide to AI‑based enterprise search for 2025
  17. [17] The AI Enterprise Search Guide for IT and Knowledge Leaders
  18. [18] Anthropic's MCP: Long-Context LLM Integration - DZone
  19. [19] The 2026-07-28 MCP Specification Release Candidate | Model Context Protocol Blog
  20. [20] Model Context Protocol (MCP) on Windows overview | Microsoft Learn
  21. [21] Model Context Protocol (MCP): A Developer’s Guide to Long-Context LLM Integration – SQLServerCentral
  22. [22] Model Context Protocol (MCP) explained: A practical technical overview for developers and architects
  23. [23] What is the Model Context Protocol (MCP)? | Databricks
  24. [24] Model Context Protocol (MCP) and Connectors: A Primer
  25. [25] What Is Model Context Protocol (MCP)? A Quick Start Guide.
  26. [26] The 2026-07-28 MCP Specification: A Stateless, Extensible Future | MCP Servers
  27. [27] David Soria Parra on X: "The release candidate for MCP 2026-07-28 is out. The protocol is now stateless: no handshake, no session id, any request can hit any server instance. Plus extensions as first-class (MCP Apps, Tasks), auth hardening, and a proper deprecation policy so we don't have to do this" / X
  28. [28] The MCP 2026-07-28 Release, Explained for Enterprise Teams
  29. [29] Beta SDKs for the 2026-07-28 MCP Spec Release Candidate Are Here | Model Context Protocol Blog
  30. [30] mcp-for-beginners/01-CoreConcepts/mcp-2026-07-28-release-candidate.md at main · microsoft/mcp-for-beginners
  31. [31] Model Context Protocol Specification Version Timeline - Version-by-Version Changes and Adoption Milestones | hidekazu-konishi.com
  32. [32] MCP 2026-07-28 spec: every breaking change, with fixes · Stacktree
  33. [33] Redis Iris: The Memory Layer Enterprise AI Agents Need — Enterprise DNA
  34. [34] AI Agents News — Week of August 28, 2026 (Daily Updates)
  35. [35] Best AI agent memory tools in 2026 - Articles - Braintrust
  36. [36] The State of AI Agent Memory in 2026: What the Research Actually Shows | by Vektor Memory | Medium
  37. [37] Best AI Agent Memory Systems in 2026: 8 Frameworks Compared
  38. [38] Types of AI Agent Memory: Retrieval vs Memory | Datafortune
  39. [39] Context architecture is replacing RAG as agentic AI pushes enterprise retrieval to its limits | VentureBeat
  40. [40] State of AI Agent Memory 2026: Benchmarks & Trends Report
  41. [41] Scaling AI Agent Infrastructure with the MCP Stateless updates - Google Developers Blog
  42. [42] MCP Just Went Stateless — What the 2026 Spec Changes About Scaling on App Service | Microsoft Community Hub
  43. [43] The 2026-07-28 Specification | Model Context Protocol Blog
  44. [44] MCP Just Went Stateless: What Changes for Your Servers
  45. [45] MCP Goes Stateless: What the 2026 Release Candidate ...
  46. [46] MCP 2026-07-28: Stateless Spec for AI Agents | BOVO Digital
  47. [47] The next generation of MCP | Cloudflare Blog
  48. [48] The 11 Best Vector Database Providers (August 2026): Features, Tradeoffs, and Use Cases | Mastra Articles
  49. [49] On-device vector databases in 2026 - AI
  50. [50] Are Vector Databases Still Relevant in 2026
  51. [51] The 2nd Workshop on Vector Databases (VecDB)
  52. [52] What's Changing in Vector Databases in 2026 - DEV Community
  53. [53] Vector Search Database: News & 2026 Guide
  54. [54] Refonte Learning : The Vector Database Shakeout: Why Postgres Keeps Winning in 2026
  55. [55] Vector Databases: The Engine Behind Semantic Search and LLM Retrieval Augmented Generation
  56. [56] Best AI Memory Systems in 2026: Why the Future Belongs to Agentic Memory OS - EverMind AI Long-Term Memory System Updates & Breakthroughs | EverMind Blog
  57. [57] AI Memory Benchmarks 2026: LoCoMo, LongMemEval & BEAM
  58. [58] Introducing STATE-Bench: A benchmark for AI agent memory | Microsoft Open Source Blog
  59. [59] GitHub - HUST-AI-HYZ/MemoryAgentBench: Open source code for ICLR 2026 Paper: Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions · GitHub
  60. [60] Evaluating Memory in LLM Agents via Incremental Multi- ...
  61. [61] MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
  62. [62] MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
  63. [63] License: CC BY 4.0

Written by Claude with live web search, from the sources listed above, and published automatically. Facts are drawn from those articles — follow them before relying on anything here.