Memora 基準評估智能體長期記憶，發現記憶過時問題

ChatGPT, Claude and Gemini all ship memory features now, and the pitch is the same: an assistant that knows you. Your preferences, your projects, your goals, carried across conversations.

OpenAI's own memory FAQ admits the awkward part. It says the previous saved-memories system "often became stale" and could hold contradicting memories at the same time. That is exactly the version of the problem a new benchmark called Memora is built to measure.

The paper behind it, "From Recall to Forgetting" (Uddin et al., Arizona State, University of Arizona, Genies, Findings of ACL 2026), argues that the way we test agent memory has a blind spot: we test recall, while in the authors' framing, real long-term memory is mostly revision.

The blind spot in how memory gets tested

Look at how existing long-term memory benchmarks are built:

• In LoCoMo, 94% of evaluation questions can be answered with evidence from at most two past sessions. In LongMemEval it is 85%. Across the major benchmarks, the average question needs about one session of history.
• Updates barely exist. LongMemEval allows at most two knowledge updates before evaluation, and PersonaMem handles changes across no more than three sessions.

So most memory scores measure something narrow: can the system dig up a fact that was stated once and never changed? Real use does not look like that. People correct themselves, change their minds, finish projects, cancel plans. Information gets updated, withdrawn and replaced, over weeks and months. A memory system that never forgets is not automatically impressive. It can be a liability with good recall.

What Memora does differently

Memora simulates 10 professional personas (engineers, researchers, designers, executives) across three timeline settings the authors call weekly, monthly and quarterly. Those labels refer to the span of the simulated history: about 155, 615 and 1,991 sessions per persona respectively, at roughly 16 turns per session.

The important part is what happens inside those sessions. In the quarterly setting, a persona's memory sees about 1,171 memory operations on average: 63% additions, 18% updates, 19% deletions. Preferences drift, activities repeat, goals progress. Every change is tracked in an explicit memory trace, so the benchmark knows exactly which facts are current and which are stale at question time.

Three task types get evaluated:

• Remembering: recall previously discussed facts.
• Reasoning: combine several memories to derive an answer (like computing budget status from logged expenses).
• Recommending: suggest things that match the user's current preferences, not their old ones.

Scoring uses a metric the authors call FAMA (Forgetting-Aware Memory Accuracy). Standard accuracy asks: did the response include the right information? FAMA also asks: did it avoid the outdated information? Each question comes with "must include" criteria and "must exclude" criteria, and using an invalidated memory costs you. Three LLM judges vote on each criterion, with 88.3% average agreement against human annotators.

They tested two setups. Four LLMs (GPT-5.2, Claude Sonnet 4.5, Gemini 3 Pro Preview, Qwen3-32B) reading conversation history directly, with and without reasoning tokens. And six dedicated memory agents (A-Mem, LangMem, Mem0, MemoBase, MemoryOS, Nemori) that store and retrieve memories across sessions, all answering through the same GPT-4o-mini backend. That last detail matters for how you read the results: the agents run on a smaller model than the LLMs reading raw history, so this is memory machinery on a fixed, modest backend, not a controlled memory-versus-no-memory comparison at equal model strength.

What they found

One note on the numbers below: the authors sum per-question FAMA across all three timeline settings, so the aggregated scores run to 300, not 100. Read them as relative scores, not percentages.

In this benchmark, memory systems are clearly better at remembering. On aggregated FAMA, memory agents nearly double the LLMs on recall tasks: 119.5 versus 65.7. Explicit memory stores do their core job.

LLMs match them on recommending anyway. 144.7 to 153.3 for plain LLMs against 138.4 for memory agents. The authors' explanation: recommendations tolerate approximate memory, because you can guess plausible preferences from persona cues without retrieving anything precise. So in this study, the task where memory should matter most is the one where it added the least.

Reasoning over memory is weak everywhere. Best average is 27.6, from the memory agents. LLMs sit at 12.4 to 13.9. In the authors' manual error analysis, every sampled reasoning error traced back to partial retrieval: the system had some of the pieces and answered anyway. One example from the paper: after logging several expenses against a monthly budget, agents said "within budget" instead of computing the remainder. Reasoning tokens made little difference.

The forgetting penalty grows as memory scales. This is the finding worth sitting with. For LLMs reading raw history, the FAMA penalty from stale-memory use shrinks on longer timelines (32.6 points in the weekly setting, 17.8 in the quarterly one), but for an unflattering reason: on long timelines the old conversations fall out of the context window entirely, so the model physically cannot reuse them. Memory agents show the opposite trend. Their penalty grows from 18.2 to 29.5. The more history the memory store accumulates, the more the agent leans on facts that stopped being true.

This changes rankings. In the monthly setting, MemoryOS (112.8) and A-Mem (112.0) both beat Nemori (105.4) on plain memory accuracy. But Nemori's forgetting penalty is much smaller (15.4 versus 28.4 and 29.5), so once FAMA counts stale memory, Nemori jumps ahead of both. In this result, the "worse" memory system gives better answers because it is better at letting go.

The error analysis makes it concrete. Of the recommendation errors the authors sampled, 64% came from outdated memory that was never invalidated. The paper's example: a user preferred non-fiction, later shifted to contemporary fiction, and the agent recommended a historical biography. It remembered. It just remembered the wrong version of the person.

What this means if you are building with memory

None of this is a reason to skip memory. It is a map of where the hard part actually is.

1. Treat memory writes as versions, not appends. An update should supersede the old fact, and retrieval should prefer the current state. A flat, ever-growing list of "facts about the user" is exactly the shape this benchmark punishes.
2. Test with corrections, not just recalls. If your eval set has no updates or deletions, you are measuring the easy half. Add "the user changed their mind" cases and check what the agent does with both versions.
3. Score the exclusions. FAMA's core trick is cheap to copy: for every eval question, write down what the answer must include and what it must not bring up. Stale-memory use is invisible until you penalize it.
4. Do not expect a memory layer to fix reasoning. Retrieval and consolidation came apart in every system tested, and the gap between them is the weakest point.

The honest caveats

The conversations are simulated by LLMs, not collected from real users. The authors argue this makes the benchmark a lower bound: real interaction adds ambiguity and contradiction, so a system that fails here will likely fail harder in the wild. Fair, but still synthetic.

As noted above, all six memory agents answered through GPT-4o-mini, so the agent numbers reflect the memory machinery more than the model. The scoring runs through LLM judges, though with three-way majority voting and a human agreement study behind it. Ten personas and three memory types (preferences, activities, goals) is also a narrow slice of what long-term memory could mean.

Even with all that, the direction of the result is hard to wave away: everyone is racing to give agents more memory, and Memora's results suggest that more memory, badly maintained, makes answers worse.

Sources: Uddin, Shubham, Blanco, Baral, Wang, "From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents", Findings of ACL 2026. Paper: https://aclanthology.org/2026.findings-acl.1337/ (preprint: https://arxiv.org/abs/2604.20006). Code and data: https://github.com/geniesinc/Memora. Product memory docs: https://help.openai.com/en/articles/8590148-memory-faq , https://support.claude.com/en/articles/11817273-use-claude-s-chat-search-and-memory-to-build-on-previous-context , https://support.google.com/gemini/answer/16598469

Written with AI assistance.