AI 助手記住事實仍答錯：可靠記憶為何不止靠嵌入向量、長上下文或更高檢索分數

Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong.

Why reliable AI memory needs more than embeddings, longer context, or a better retrieval score.

An assistant remembers that Sarah works at Acme.

Then Sarah changes jobs.

A month later, you ask:

“Where was Sarah working when we met?”

The assistant confidently answers with her new employer.

It retrieved a real fact. It identified the right person. It may even have selected the most recently updated record.

And it still answered the wrong question.

This is a useful way to think about AI memory failures: sometimes the problem is not forgetting. It is remembering something without preserving the conditions under which it is relevant.

For an engineering design, I would treat validity as a first-class concern—not metadata added after retrieval.

One Sentence Can Contain Two Different Timelines

Consider this message:

“I moved to Pune in March.”

Suppose the user says it in June.

There are at least two distinct times to represent:

• When the move occurred.
• When the system learned about the move.

These timestamps answer different questions.

“Where did I live in April?” asks about the world.

“What did you know about my location in April?” asks about the system’s knowledge at that point.

Treating both as a single message timestamp loses that distinction.

This is the motivation behind a bitemporal representation: separate valid time—when a fact held in the world—from transaction or ingestion time—when the system recorded it. Graphiti’s architecture explicitly distinguishes these timelines and tracks validity and record lifecycle timestamps. arxiv

This is not a claim that every assistant needs a temporal graph database. It is a claim that the data model needs to distinguish questions the product promises to answer.

“Outdated” Does Not Mean “Useless”

Imagine these statements arriving over time:

“I work at Acme.”

“I’m joining Nova next month.”

“My start date moved back two weeks.”

“Actually, I decided to stay at Acme.”

A memory pipeline that simply overwrites current_employer risks collapsing several different concepts:

• An existing state.
• An intention.
• A revised plan.
• A canceled transition.

My proposed design rule is simple:

Do not turn every newly extracted sentence into an immediately valid replacement fact.

A plan is not an event. A correction is not always a new state. A later message can change how an earlier message should be interpreted.

Temporal memory systems such as Graphiti preserve superseded facts through invalidation rather than simply deleting them, allowing current and historical relationships to remain distinguishable. getzep

But preserving history is only part of the design. The reader must also choose the right historical state for the question.

Store Evidence Separately From Current Beliefs

I would separate three layers in a memory implementation.

1. Source evidence

What was actually said?

Keep the source message, speaker, timestamp, and enough surrounding context to interpret it.

2. Extracted assertions

What does the system think the source claims?

For example:

{
  "subject_id": "person_sarah_01",
  "predicate": "employed_by",
  "object_id": "company_acme_01",
  "assertion_type": "reported_state",
  "recorded_at": "2026-06-10",
  "valid_from": null,
  "valid_to": null,
  "source_message_id": "message_184"
}

This is an illustrative schema, not a complete temporal database design.

The null dates matter. If the source does not establish when employment started, the extractor should not invent a start date.

3. Derived views

What should the assistant currently believe?

A current-state view can select the assertions relevant to “Where does Sarah work now?”

A historical view can select evidence relevant to “Where did Sarah work when we met?”

These views should point back to the underlying evidence.

The architectural goal is to make interpretations revisable without losing the information used to produce them.

Identity Needs the Same Discipline

Now introduce two people named Sarah.

One is your colleague. The other is your sister.

A name alone cannot distinguish them in this example. But “name plus relationship” is not necessarily a permanent identifier either: colleagues become friends, managers change, and relationships can overlap.

I would use stable internal entity IDs, with names and relationships stored as evidence-backed attributes.

For ambiguous references, the write path should have more than two options.

Instead of only:

Create entity
or
Merge into existing entity

allow:

Create entity
Link with sufficient evidence
Hold as unresolved
Ask for clarification

A system that can preserve uncertainty does not have to turn every ambiguous mention into a confident merge.

Retrieval Should Select a State, Not Just a Sentence

Consider four questions:

1. Where does Sarah work?
2. Where did Sarah work last spring?
3. What did Sarah say about changing jobs?
4. What did we believe about Sarah’s employer before her correction?

They concern the same person and topic, but they ask for different evidence.

For this design, I would represent query intent explicitly enough to distinguish:

• Current state.
• State at a specified time.
• Events or changes.
• Previously recorded beliefs.

The retrieval pipeline could then look like this:

Question
   ↓
Entity resolution
   ↓
Temporal and question-intent interpretation
   ↓
Candidate retrieval
   ↓
Validity and provenance checks
   ↓
Evidence selection
   ↓
Answer or abstention

This is a proposed decomposition, not an argument that every box requires a separate model call.

The important part is that “most similar” and “appropriate evidence” are not treated as interchangeable.

Cross-References Should Not Become Conflicting Copies

Suppose a fact concerns both Sarah and Acme.

It should be discoverable through either entity.

For a structured implementation, I would prefer one canonical fact record referenced by both entity indexes rather than two independently editable copies:

Sarah index ──┐
              ├── fact_271
Acme index ───┘

That gives corrections one authoritative target.

Materialized copies may still be useful for performance, but then their consistency needs to be an explicit engineering responsibility.

A chapter metaphor can make navigation intuitive. It should not accidentally dictate an update model that is difficult to maintain.

A Good Retrieval Score Can Hide a Broken Write Path

Imagine a memory system with three stages:

Conversation → Extraction → Filing → Retrieval

A wrong answer could originate anywhere:

• The extractor missed the relevant statement.
• The entity resolver chose the wrong person.
• The fact was assigned to the wrong attribute.
• A correction was applied too broadly.
• Retrieval selected the wrong time period.

Improving the retriever will not reveal which of these happened.

Before choosing the next model to train, I would run an ablation:

Keep the retrieval procedure fixed.

Condition A is a reference condition for retrieval with verified inputs—not a theoretical upper bound.

The other conditions help investigate which upstream errors are responsible for the observed failures.

The evaluation should report those failures separately, rather than reducing every error to “memory recall.”

Test the Questions the System Should Refuse to Answer

A useful test set should contain more than questions with known, neatly stored answers.

I would include:

• An unknown start date.
• Two unresolved people with the same name.
• A job offer that was never accepted.
• A correction with no stated effective date.
• Conflicting statements from different speakers.
• A question whose answer was never provided.

For each case, define acceptable behavior in advance.

Sometimes the correct output is a fact.

Sometimes it is a qualified answer.

Sometimes it is:

“I know Sarah changed employers, but I don’t have enough information to determine where she worked on that date.”

An evaluation that always requires a factual answer cannot tell you whether the system handles missing evidence appropriately.

Measure the Assistant, Not Only the Memory Index

Retrieval evaluation is useful, but it does not automatically establish the performance of the complete assistant.

For a proposed system, I would track at least three groups of measurements.

Write-path measurements

• Extracted-fact accuracy.
• Entity assignment accuracy.
• Attribute assignment accuracy.
• Correction handling.
• Update latency.

Read-path measurements

• Relevant evidence retrieval.
• Correct temporal selection.
• Entity-confusion errors.
• Unsupported-answer frequency.
• Answerable versus unanswerable behavior.

End-to-end measurements

• Final answer quality.
• Tool calls per question.
• Retrieved context tokens.
• Response latency.
• Recovery after incorrect writes.

A selective reader may reduce context size, but that remains an empirical question until the full reading loop is integrated and measured.

Likewise, an extra retrieval step might save prompt tokens while adding latency. The relevant trade-off belongs in the end-to-end results, not only in an architectural diagram.

Memory Is a Data-Lifecycle Problem

Semantic retrieval remains useful. It can help locate evidence when the user’s wording differs from the stored text.

But my design recommendation is to place it inside a broader lifecycle:

Capture
→ Interpret
→ Resolve identity
→ Record provenance
→ Establish validity
→ Retrieve
→ Verify
→ Correct

Existing systems already address parts of this lifecycle. Graphiti documents temporal invalidation and historical relationships; Mem0 exposes a memory-history interface for tracking how an individual memory has changed. Neither should be reduced to “just a pile of embeddings.” help.getzep

The useful engineering question is therefore not simply:

“How do we give an assistant memory?”

It is:

“What evidence does this assistant preserve, how does it revise that evidence, and under what conditions is it allowed to use it?”

An assistant that remembers yesterday’s fact is useful.

An assistant that can explain whether the fact is still true—and recognize when it cannot tell—is a more demanding system to build.