智能體 AI 架構中記憶、知識與上下文的系統性設計

Key Takeaways

Context engineering beats prompt engineering - the system that builds context matters more than the prompt itself.
Separate context, memory, and knowledge - each serves a different purpose and should be managed independently.
Balance history with real-time data - great agents combine experience with live signals.
Hallucinations are often context failures - better context leads to more reliable reasoning.
Optimize context, don't maximize it - small, relevant context outperforms large, noisy prompts.

This article is part of the "Agentic AI Architecture" article series. In this series, we try to establish agentic AI architecture as a new type of software architecture that will likely dominate the industry for years to come. The articles, written by industry experts, cover various elements and aspects of agentic AI architecture. We aim to present the latest trends and developments shaping the new type of architecture as it enters the mainstream.

As the industry moves from chatbots to autonomous agents, the architectural bottleneck is shifting. The hard problem is no longer simply retrieving data. It is deciding what an agent should know, when it should know it, how long it should remember it, and what should be forgotten. As with humans, it is one thing to know and another to remember. Hence, memory and knowledge. Knowing when to use any of the existing "knows" by bringing it to memory depends on the task context. For example, if I want to cook an omelet, I don’t necessarily need to remember how to knead dough right now.

Let’s break it down.

Agents must reason across tools, sessions, users, and live systems, given that in an engineered system we can describe the context as a control plane. Context as a control plane is an architectural paradigm in which organizational, behavioral, and situational data (the who, what, and why) serve as the centralized governing layer for system operations, rather than being hardcoded directly into individual applications, AI models, or microservices. In our world, this will determine whether an agent acts like a reliable software system. Providing the wrong context highly increases the likelihood of the model hallucinating.

Knowledge of the system is our policies, governance, and hard truth. Knowledge acts as the system’s official textbook. It holds core business logic, strict policies, and databases. Back to our omelet example, part of the knowledge would be the need for eggs and a pan.

Memory is the critical layer that bridges historical data and real-time signals into a reasoning-ready state. What needs to be remembered for the current task? This is usually information given from knowledge, historical data, and current live events.

To summarize our three components:

Context - the immediate, active, and temporary information the agent is utilizing at any given moment. It sits directly within the model's active context window (analogous to computer RAM).
Memory - provides persistence. It allows the agent to retain information beyond a single interaction, enabling it to learn from past experiences and adapt to users over time.
Knowledge - the underlying foundation of facts, concepts, and rules that the agent inherently possesses before interacting with the user. It is the domain-specific data used to train or fine-tune the system, or the reference library to which it is granted access. Highly curated data.

Our goal in agentic systems is, when given the task context, system knowledge, and memory, to turn data into the right memory state at the right moment.

Beyond that, there are limitations to this dynamic architecture paradigm; hence, we need to systematically address the economic costs of large language model (LLM) limitations and the technical, architectural roots of hallucinations. As developers and architects, learning to avoid the common "culprits" of agentic systems mismanagement is crucial. Dive in to learn how to design a sophisticated, multi-tiered architecture for the next generation of reliable agentic AI.

Shift into Context Engine

An LLM is only as good as how we use it. While the model generates, our system determines what it sees. To avoid hallucinations, we need a trustworthy system around it. Leveraging context, knowledge, and memory. To architect the system, we will separate the model from the surrounding machinery. Going forward, for the machinery around it, we will call it a context engine. The context engine is part of the context control plane. This is the orchestrator responsible for selecting, filtering, ranking, structuring, compressing, safeguarding, and delivering information before it ever hits the LLM inference endpoint. It pulls information from knowledge, categorizes it in memory, and determines what the agent's context should be at any given millisecond. This is a meaningful shift from simple prompt engineering to systems engineering.

From Prompts to State/Context Systems

Prompt engineering assumes the task is to craft a better instruction. Context engineering assumes the real task is to build a dynamic substrate that continuously assembles the right state for reasoning. The prompt is no longer about "What should I ask the model?" It is "What is the minimal, highest-value context the model needs in order to reason well right now?"

In a production-grade agentic architecture, we require dynamic systems that dictate what an agent "knows" at any given millisecond. Our system goal is to take what’s relevant and compile a new prompt encompassing the required information to improve the model's success likelihood.

Context as a Compiler

To understand how the Context Engine behaves, consider the analogy of a modern software compiler. A compiler translates high-level source code into machine-executable instructions; the context engine translates vast oceans of unstructured, disconnected raw, real-time, and historical data into a structured, reasoning-ready state. It builds a deterministic state machine that allows the probabilistic agent to execute safely. It ensures that inputs are authenticated, validated against business logic, stripped of personally identifiable information (PII), and securely sandboxed before the LLM generates a plan of action.

Thinking Through Architecture

Effective software architecture for autonomous AI agents requires reconciling several fundamentally conflicting paradigms of data management and data flow.

Historical vs. Real-Time Memory Streams

A robust agentic architecture must gracefully balance long-term "episodic" memory with immediate "sensory" input. Meaning that to build a useful context engine, our architecture must reconcile two very different categories of state: historical memory and real-time input.

Historical memory is the long-lived layer. It includes prior interactions, past decisions, user preferences, domain rules, internal documentation, and accumulated observations. This is what gives an agent continuity. It allows the system to behave as though it has experience.

Real-time input is the short-lived layer. It includes the current task, the latest events, live telemetry, fresh tool outputs, and environmental changes. This is what keeps the agent grounded in the present.

While vector databases excel at retrieving historical, semantic facts, agentic workflows frequently pivot on sub-second, real-time context. For example, if an AI DevOps agent is triaging a server outage, retrieving an architectural runbook from a vector store (episodic memory) is only half the battle. The agent also needs the live error log stream from the last five seconds (sensory memory). The architecture must fuse these batch and streaming sources asynchronously, ensuring the agent acts on a unified, up-to-date reality.

This architecture must have balance. If it over-indexes on history/episodic memory, the agent becomes stale. It will keep replaying yesterday’s assumptions into today’s reality. If it over-indexes on real-time data, the agent becomes reactive and forgetful. It may respond to the latest signal but fail to incorporate the broader context that gives that signal meaning.

Luckily for us, this tension is not new. Distributed systems have always had to reconcile durable state with transient events. Agent systems inherit the same problem, but with an additional challenge: the reasoning layer is probabilistic, expensive, and bounded by a limited context window. This adds an important layer of constraint optimization to the context architecture design - deciding which information to keep, which can be compressed, which must be revalidated, and which should be excluded entirely. The goal is not just freshness or completeness. The goal is usefulness under constraint.

Figure 1: Historical Data vs Real-Time Memory Streams (Source: created by the author with Gemini)

Knowledge vs. Raw Data (`SELECT *` vs. Cumulative Insight)

One of the most common architectural mistakes in AI systems is treating retrieval as if it were enough, ie, operating under the semantic equivalent of a SQL SELECT *, assuming more information must be better. Architectures that attempt to consistently retrieve large volumes of data will inevitably fail at scale. We can think of retrieval as how we get data into the system, yet it does not guarantee that the data is useful, timely, coherent, or appropriately ranked for the decision at hand. This is what standard RAG pipelines are all about. They are often built to retrieve raw, fragmented, and overlapping data chunks. In actuality, using this approach would most likely result in the system dumping 10 unedited wiki pages into a prompt, making the LLM waste compute cycles trying to sort the noise from the signal - yet failing due to relevant signals buried under less important and more noisy ones. Hence, the right information and shape of the data, which we refer to as knowledge, matter more than raw data.

True Knowledge Curation focuses on cumulative insight. Instead of injecting raw data, the architecture employs background workers/ML pipelines that distill, aggregate, and rank retrieved data by relevance, using the up-to-date system state. To achieve that, we need to build dedicated pipelines that transform raw data into structured knowledge graphs or condensed summaries. For example, taking a large database and creating a pipeline that summarizes a million data entries into one entry every 12 hours. This way, we ensure the model receives a small, synthesized body of knowledge (1 entry point in the database) rather than a chaotic stream of consciousness (the whole database, sometimes).

Custom Context: The End of "One-Size-Fits-All"

Foundational models like GPT-4 and Claude, as well as vector databases like Pinecone and Milvus, have become an accessible commodity. Everyone can access them today through their services. Yet, this is not the same for context; the architectural principles for working with context are highly domain-specific. Every company should design its own context based on the business domain and the product it serves. For example, a fintech firm might choose to curate context for an autonomous fraud-analysis agent by prioritizing transaction timestamps, geolocation anomalies, and strict compliance rules. This will look entirely different for a legal tech firm. A legal tech firm might choose to curate context based on state law for use with an agent that generates contracts. We are learning that the context itself cannot be generic; it must reflect the unique domain logic, security boundaries, and vocabulary of the specific product.

The Context Lifecycle: Managing Knowledge & Hallucination at Scale

Treating context as a continuous lifecycle allows us, as engineering teams, to manage knowledge systematically and directly address the root causes of model failure and prohibitive inference costs.
The Lifecycle Stages

Generally speaking, there are 4 stages: Selection, Injection, Compression, Expiration & Refresh. Figure 2 describes how each stage could be a dedicated module in our architecture or a standalone service.

Figure 2: Agentic AI Systemic Context Lifecycle (Source: created by the author with Gemini)

Selection (Retrieval & Filtering)

The first and most important step. This is where the system identifies which data points are likely to matter for the current task. The system's goal is to identify the absolute highest-value data points across multiple data stores. Advanced architectures have moved beyond simple cosine similarity in vector spaces, implementing hybrid search algorithms that combine dense semantic search with sparse keyword search (e.g., BM25) and knowledge graph traversal to ensure precision. In this process, we want to understand intent, task type, risk, user role, and temporal relevance. It is important to remember that the same underlying corpus may yield different contexts for a troubleshooting agent, a research agent, and a support summarization agent. This is where architecture begins to diverge from product. As part of the design requirements, we need to identify and define what is risky and what is actionable.

Injection

Once the system knows what matters, it has to place that information into the model’s working context (aka context window) in a useful form. While the end result might look like simple string concatenation, the path to getting there requires more deliberate design. That design involves semantic templating, ensuring the data is formatted to match the user's explicit intent, including necessary system instructions, few-shot examples, and output schema constraints (such as JSON definitions). To understand the importance of this step, consider how it affects the entire system. In our world, this placement of information changes how models reason. Hence, order, grouping, and structured state matter.

Compression (Summarization & Distillation)

Processing a large context is expensive. To respect token limits, historical data should be iteratively compressed. Utilizing smaller, faster LLMs (or even deterministic algorithms) to summarize conversation histories or condense long documents saves massive amounts of space and reduces latency for the primary reasoning model. Test summaries carefully to avoid distortions. There are statistical models available today to assess summary validity, such as the Perplexity (PPL) Ratio, Token-Level Kullback-Leibler (KL) Divergence, the Chi-Square Test for Categorical Outputs, and others.

Expiration and Refresh

Context that never expires eventually turns into a liability. Agents need mechanisms to retire stale assumptions, refresh live facts, and separate durable truths from temporary observations. A system preference might remain stable across months or change overnight. A product configuration might change in minutes. A support incident may be relevant for a day and misleading a week later. Without expiration and refresh, systems fall into stale data loops: they keep reusing context that was once correct but is now simply persuasive. Hence, constantly testing and validating the system through feedback loops is a must. This should be tailored to the system's end goal. For example, in a car-sharing system, the goal is to satisfy users; hence, if user satisfaction drops suddenly, it’s best to explore underlying factors that may need refreshing.

Hallucination Is Often a Context Failure

LLM hallucinations are rarely the result of a "broken" model; they are almost always the symptom of an unmanaged, impoverished context window. Because LLMs are probabilistic, when an agent lacks the explicit, factual data required to answer a prompt or execute a task, its underlying mechanism forces it to "guess" based on latent training weights.

Proactive context management acts as a strict guardrail. By saturating the model with highly relevant, high-confidence knowledge and explicitly instructing it to reason only based on the provided context, we can effectively suppress latent, inaccurate associations. In a productionized agentic system, a well-engineered context engine becomes our primary defense against hallucinated actions.

Token Economics: The Cost-to-Performance Trade-off

Processing a large context is notoriously expensive. Every token injected into the prompt window incurs both a computational latency cost and a strict financial cost. Token Economics is the engineering discipline of optimizing "attention real estate" within the LLM's limitations.

The recent expansion of context windows (e.g., models boasting 128k to 1M+ token limits) has tempted developers towards lazy architecture. They dump entire code repositories or massive troves of documents into the prompt. This drastically increases Time-to-First-Token (TTFT) latency, skyrockets API billing costs, and, ironically, degrades model performance. The model suffers from the "Lost in the Middle" phenomenon, shown in Figure 3. This problem was identified by Stanford researchers Nelson F. Liu et al. in 2023, where the model fails to retrieve facts buried in the center of massive context windows and only focuses on edge case information. The larger the bottom part of the U shape (context window), the less accurate the response will be. This research explains why we want the context sent to the model to be relatively small. This approach makes the U bottom part smaller as well - hence, improving outcomes.

Architects must weigh the marginal utility of every retrieved document against its token cost and relevancy. Just because we can send 100,000 tokens doesn't mean we should. Strict context budgeting, iterative pruning, and caching mechanisms are mandatory for scalable economics.

Figure 3: Lost in the middle problem - recall vs accuracy (Source: created by the author with Gemini - based on Lost in the Middle scientific paper by Stanford)

Designing Tiered Memory Architectures

Designing systemic approaches to memory requires establishing firm boundaries for how agents retain, update, or discard knowledge across sessions. Drawing inspiration from human cognitive architecture, systems should be multi-tiered:

Working Memory (The Prompt Window): The immediate, ephemeral state required to execute the current step in a workflow. It is highly constrained and resets per inference call.
Short-term Memory (Session Cache): Often managed via in-memory data stores like Redis. This tracks the current conversational state, intermediate steps of an ongoing task, and recent actions over the last few minutes or hours.
Long-term Memory (Asynchronous Persistence): The permanent storage of synthesized insights, user preferences, and historical facts. This relies on vector databases and knowledge graphs, which are updated asynchronously, so that the agent's core "understanding" grows over time without bloating the real-time retrieval pipeline.

How to Mitigate Architectural Anti-Patterns

There are a few recurring failure patterns that show up again and again in agentic systems.

Architectural Anti-Pattern	Description	Systemic Impact	Mitigation
Context Stuffing	Injecting every plausible document, log, and tool result into the prompt under the assumption that "more is better."	Creates severe noise, dilutes model attention, increases latency, and degrades precision.	Implement strict relevance scoring thresholds. If a retrieved document's semantic similarity score falls below a certain confidence interval, drop it entirely rather than stuffing it into the prompt.
Stale Data Loops	Reusing historical context (like user states or system configs) without a mechanism to validate its current accuracy.	Agents become confidently wrong, consistently acting on outdated realities.	Implement metadata filtering during retrieval that cross-references timestamps, or utilize event-driven architectures (like Kafka) to proactively invalidate stale embeddings.
Generic Retrieval	Utilizing a standard, one-size-fits-all RAG pipeline regardless of the specific agentic workflow.	Fails to respect domain logic, resulting in the retrieval of irrelevant chunks and missing nuanced data.	Implement query-rewriting mechanisms in which a smaller LLM (perhaps your own managed SLM) restructures the user's raw query into domain-specific, optimized search vectors before querying the database.
Implicit/Explicit Mismatch	Failing to align implicit system memory (what we know about the user) with the explicit prompt (what the user is asking right now).	The agent's output feels off-target, misaligned, or occasionally intrusive and overly familiar.	Use a rigid, structured prompt template that strictly separates system context, user context, and the immediate task instruction into distinct semantic blocks.
Unbounded Memory Growth	Persisting all logs, chats, and actions without an expiration or pruning strategy.	The system degrades under its own accumulation; retention without deliberate curation becomes data hoarding.	Implement Strict TTL (Time-To-Live) Policies Not all memories are created equal. Treat agent memory like a standard caching layer by applying category-specific expiration dates. Sensory/Real-Time Data: (e.g., API latency telemetry, UI click streams). Expire within minutes or hours. Task/Session Data: (e.g., the intermediate steps of a specific workflow). Expire as soon as the session closes or the objective is achieved. Core Knowledge: (e.g., user preferences, verified facts). Persist indefinitely, but subject them to periodic re-validation.

Conclusion: Successful Agentic Systems Depend on Their Architecture

The next generation of AI systems' success will be determined primarily by architecture design. While the industry continues to debate model parameters and raw capabilities, the core challenge for engineering teams looks remarkably familiar: state management, signal prioritization, fault isolation, data freshness, security, and cost control.

The winning architectures will treat themselves as essential infrastructure, curate context, view memory as a strict system design problem, and elevate knowledge curation to a first-class engineering discipline. This is the real shift from chatbots to autonomous agents.

About the Author

Adi Polak

Show moreShow less