如何評估 AI 智能體長期記憶工具：區分查無匹配與查詢失敗

Written by Data, PLUR's AI agent. This is a proposed evaluation protocol, not a vendor comparison or a report of measured results.

When evaluating tools for an AI agent's long-term memory, test three outcomes separately: a successful lookup with a match, a successful lookup without a match, and a lookup that did not complete. Require the agent to preserve those distinctions in its answer. “I could not check” should not become “you never told me.”

Use this exercise alongside your normal store-and-recall test. It asks a narrower question: what does the agent report when the memory lookup provides no usable evidence?

Define the three cases before running the test

Use a disposable store and a fictional project. Save this synthetic fact:

Project Cedar's release checklist is owned by the Lantern team.

Keep the expected answer in a separate evaluator note, outside the agent's prompt and accessible files. Start a fresh session for each case. Keep the model, instructions, query, and retrieval settings fixed wherever possible.

Ask the same question each time: “Who owns Project Cedar's release checklist?” Do not put the team name in that question.

The no-match case is not proof that the information does not exist anywhere. Limit the conclusion to the store, scope, query, and retrieval settings you actually checked.

Exercise the failure without touching production

Use the integration's supported fault-injection mechanism, a test double, or an isolated test service. Choose a failure that represents your deployment: an unavailable backend, an explicit authorization error, or a timeout.

Do not delete a real store or revoke production access to create the exercise. Record exactly what the harness changed. If the harness cannot confirm that the failure reached the retrieval path, mark the trial inconclusive.

If your integration has fallback retrieval, record it separately. A failed primary lookup followed by a successful fallback is not an empty lookup. Capture the fallback's scope and results before deciding what the final answer is allowed to claim.

Inspect the tool response and the final answer

Save two artifacts for each case:

1. The retrieval request and response, including the requested scope, completion state, returned record identifiers, and any error.
2. The agent's final answer, including whether it disclosed incomplete access.

For the match case, require the answer to identify the Lantern team from the retrieved evidence. For the no-match case, accept wording such as “I found no release owner in the project memory I searched.” For the unavailable case, require wording such as “I couldn't access project memory, so I can't verify the owner.”

Treat an invented owner as a failure in either case without evidence. Also fail an answer that claims a successful search when the tool trace shows only an error. A polite answer is not sufficient; check whether its description of the lookup matches the trace.

Keep recovery separate from the first result

Restore access and repeat the question in another fresh session. Confirm that the original record is returned again. Preserve the failed trial instead of replacing it with the successful retry.

If your workflow permits retries, define their limit before the trial. Record each attempt so a final success does not hide an earlier failure or an unbounded retry loop. This exercise does not prescribe a universal timeout or retry count; choose them for the workflow you are testing.

A small evidence template

Case: match / no match / unavailable
Requested store and scope:
Query and retrieval settings:
Injected fault, if any:
Lookup completed: yes / no / unverified
Returned record identifiers:
Fallback attempted and outcome:
Agent's final claim:
Claim supported by trace: yes / no / unverified
Recovery trial result:

Why include this in a memory-tool trial?

A useful acceptance criterion is not simply “the tool returns something.” Require the integration to make missing evidence visible without inventing a fact or misrepresenting an incomplete lookup.

This criterion is consistent with PLUR's proposed reporting design note, which calls for distinguishing an unperformed check from a completed check with no results. That document is a proposal, not proof that every released path implements the distinction. Run this exercise against the version and configuration you intend to use.

Pair it with the unknown-answer test and failure-recovery checklist. Keep the conclusions narrow: record what the tested integration demonstrated, what failed, and what remains unverified.