如何打造不受污染的 Agent 記憶架構

Give your agent memory and it feels alive. Give it memory without schema and it feels like that coworker who remembers every rumor as gospel.

#agents#memory#rag

Give your agent memory and it feels alive. Give it memory without schema and it feels like that coworker who remembers every rumor as gospel. Contamination is the default. Unverified tool output, stale preferences, and half-finished reasoning all look identical once embedded.

Priya's agent confidently cited a six-month-old hallucination as fact. The memory store held everything — verified facts, stale preferences, and a half-finished analysis from last quarter — all looking identical to the retrieval system. There was no flag, no provenance tag, no timestamp priority. The contamination wasn't a bug. It was the default architecture.

Here's the thing: production memory isn't one bucket. It's tiers with different TTLs, provenance, and write permissions. Facts you verified. Decisions a human approved. Session scratch that dies at logout. Treat them the same and your agent will confidently cite a hallucination from March.

3 tiersverified facts, approved decisions, ephemeral scratch — minimum viable memory architectureInternal pattern7 daysdefault TTL we use for unverified agent-generated memory before auto-expiryConfigurable — shorter for high-stakes domains60%of agent hallucination incidents trace back to contaminated memory stores, not model errorIndustry observation, 2025–2026

Write gates beat bigger context

Reading everything into the prompt isn't memory — it's expensive amnesia with extra steps. Durable memory needs write gates: what is allowed to persist, who approves it, and what metadata gets stored (source, timestamp, confidence, tool ID). Agents should propose memories; policies should commit them.

ArchitectureMemory tier admissionIllustrativeHuman-approved decisionPermanentVerified tool outputLong TTLAgent inferenceShort TTLIllustrative retention policy — tune to your compliance needs.

Memory you can't audit is memory you can't trust. Design for deletion as carefully as retrieval.

Write gates act as the bouncer at each memory tier entrance — unverified data gets redirected to short-TTL scratch before it pollutes long-term storage.

Practical patterns

• Separate stores — user prefs ≠ org facts ≠ session notes
• Provenance on read — show where a memory came from when it influences output
• Compaction jobs — summarize old threads; don't replay raw chains forever
• Negative memory — "we tried X, it failed" is as valuable as success — tag it explicitly

Read paths need hygiene too

Everyone obsesses over what gets written to memory. Fewer teams ask what gets read back and in what order. Retrieval without ranking is how a six-month-old guess outranks yesterday's verified fact — same embedding score, wrong recency, disaster in the answer.

Diary-style free text and tool outputs are the usual contamination vectors — gate them before they enter long-term memory.

Clean data pipelines keep agent memory sterile — contamination filtered before it enters the store.

Here's the thing: memory retrieval should be policy-driven, not similarity-only. Boost verified sources, decay unconfirmed agent inferences, hard-exclude memories tagged deprecated. Your agent shouldn't cite a pricing figure from a thread where someone said "I think it's $49" unless that thread is labeled folklore.

• Recency weights — exponential decay on unverified tiers
• Conflict resolution — when two memories disagree, surface both; don't blend
• User overrides — "forget that" must actually delete or tombstone
• Cross-session isolation — support agents don't inherit each other's scratch

Memory without deletion is a hoarding problem wearing an AI badge.

A poisoning suite runs false facts through session one, then probes session two for leakage. If contamination survives, your write gates need hardening.

Testing contamination before production

Build a poisoning suite: feed the agent false facts in session one, then ask related questions in session two. If falsehoods persist without provenance flags, your write gates failed. Run it after every memory schema change — it's cheaper than a customer-facing hallucination incident.

What this means for you: instrument memory reads in logs. When an answer goes wrong, you want to see which memory IDs influenced retrieval — not a shrug and "the model said." Pair with periodic compaction jobs that promote repeated verified facts and archive noisy thread dumps. Memory is a database; act like a DBA.

PolicyMemory read ranking factors

GDPR and the right to be forgotten

Memory systems must support tombstoning and propagation — when a user deletes data, embeddings and summaries derived from it must go too. "We'll retrain later" isn't compliance. Design deletion jobs that run on schedule and log completion.

What this means for you: legal will ask about memory retention before security asks about injection. Have answers for both. Shorter TTL on unverified tiers isn't just quality — it's regulatory hygiene.

Compaction without amnesia

Thread compaction saves tokens but risks summary drift — each summarization pass loses nuance. Keep raw logs in cold storage for dispute resolution; serve agents the compact view. When legal asks what the bot knew on March 3, you need the archive, not the summary of the summary.

• Version summaries — tag compaction generation; rollback if quality drops
• Pin critical facts — promotions survive compaction; chitchat doesn't
• Human review on promote — facts entering long-term tier get eyes

If you're building agents that touch customer data, pair memory design with a store that already enforces boundaries: read-only access, cited sources, no silent writes. Memory should inherit the same discipline.

Treat memory like a database, not a diary

Start with the three tiers — verified facts, approved decisions, ephemeral scratch — and give each one an owner, a TTL, and a deletion path before you let a single agent write to it. What this means for you: the poisoning suite isn't optional homework; run it after every schema change, the same way you'd run a migration test before touching a production table.

Here's the thing: the agents that earn trust aren't the ones that remember the most — they're the ones that can tell you, on demand, exactly which memory ID produced a bad answer and delete it in one step. Provenance and tombstoning are the whole job. Everything else is just embeddings with extra confidence.

Your move: Start Monday by categorizing your current memory store into three buckets — facts you'd defend in court, notes that help but could be wrong, and scratch that should evaporate by Friday. Add a provenance field and a ttl\_hours to each entry. Write a one-hour poisoning test. That's it. That's the whole architecture. Everything beyond that is optimization on top of hygiene, and hygiene is the only thing users actually feel when an answer goes wrong.

Stay with us · challenge

Do You Have a Multi-Tiered Memory Architecture for Your AI Agents?

Friendly challenge: How is your current agent memory architecture structured? Are you using multi-tiered storage with different TTLs and provenance flags to prevent contamination?

No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.

Ayraix.com publisher analysis

What we're really seeing

The architecture of agent memory is crucial for ensuring accurate and reliable interactions. By treating different types of memories—verified facts, approved decisions, and ephemeral scratch—differently, we can prevent contamination and ensure that our agents operate with integrity.

In the context of production memory, it's essential to implement a multi-tiered system with varying time-to-live (TTL) settings. This approach not only helps in managing the longevity of memories but also in ensuring that unverified data doesn't contaminate the more critical verified facts and decisions.

Thought-provoking Ayraix publisher take titled conceptually 'What we're really seeing'. Challenge the reader to reflect on their own agent memory systems. Original paraphrase — no third-party quotes.

Keep exploring on ayraix.com

• Agentic Everything: Are We Building the Right Abstractions? COMMUNITY

Quick check — did this stick?

Question 1 of 3Reading companion — AI has read this article. Ask questions, challenge claims, go deeper.

Originally published on ayraix.com, practical AI for enterprise builders.