AI 智能體系統設計：從示範走向正式環境的 6 個重要概念

Building an AI agent demo is easy. Building one that is reliable, scalable, secure, and production-ready is much harder.

I recently watched Agentic AI System Design was HARD until I Learned these 6 Concepts and summarized six important concepts for designing production AI agents.

Let's use a customer support agent as an example. Imagine a customer asking:

"I never received my order. Please cancel it and refund me."

The agent needs to understand the request, retrieve the order, check refund policies, execute authorized actions, and respond correctly.

Here's what makes this challenging.

1. Agent Loop: Plan → Act → Observe → Repeat

Unlike a traditional LLM application that simply generates a response, an AI agent can reason, call tools, observe results, and decide what to do next.

A typical agent loop:

1. Plan: Determine what needs to happen.
2. Act: Call a tool or execute an operation.
3. Observe: Examine the result.
4. Repeat: Continue until the task is complete.

The challenge is reliability decreases as the number of steps increases.

If each step has a 90% success rate, assuming independent failures:

• 3 steps: 72.9% success
• 5 steps: 59.0% success
• 8 steps: 43.0% success

This illustrates why unnecessary complexity can make agents unreliable.

Key takeaway: Minimize unnecessary steps, validate intermediate results, and implement retries, fallback strategies, and clear stopping conditions.

2. Model Routing: Use the Right Model for Each Task

Using the most powerful LLM for every task increases cost and latency unnecessarily.

Different tasks require different levels of intelligence.

This technique is called model routing.

The objective is to balance:

Quality ↔ Cost ↔ Latency

Key takeaway: Use expensive reasoning models only when additional reasoning improves the outcome.

3. Tool Design: Treat Tools Like Production APIs

Tools allow agents to interact with external systems.

Our customer support agent might have:

• get_order_details
• search_refund_policy
• check_refund_eligibility
• issue_refund

However, adding more tools also creates more opportunities for incorrect tool selection and execution failures.

Well-designed tools should provide:

• Clear names and descriptions.
• Strict input/output schemas.
• Structured error responses.
• Authentication and authorization.
• Separation between read and write operations.

For example, retrieving an order is low-risk, while issuing a refund changes financial records.

The video also introduces MCP (Model Context Protocol), an open standard for connecting AI applications with tools and external data.

One additional production consideration is idempotency.

If a refund request succeeds but the API response times out, retrying it should not accidentally issue another refund.

Key takeaway: Let the LLM propose actions, but use backend code to validate and safely execute them.

4. Memory, State, and RAG: Give Agents the Right Context

These concepts solve different problems.

State represents the current workflow execution:

• Current step.
• Order ID.
• Approval status.
• Refund status.

Memory represents longer-term information:

• Previous conversations.
• Customer history.
• User preferences.

RAG (Retrieval-Augmented Generation) retrieves relevant external knowledge, such as company refund policies.

Instead of loading an entire policy handbook into the prompt, RAG retrieves only the relevant passages.

A typical RAG pipeline:

Documents → Chunking → Embeddings → Retrieval → Ranking → LLM

But retrieval quality matters.

If the system retrieves an outdated or incorrect policy, the LLM might confidently generate an incorrect answer.

Important RAG considerations include:

• Chunking strategy.
• Metadata filtering.
• Retrieval accuracy.
• Reranking.
• Document freshness.

Loading too much information into the context window can also introduce context rot, where irrelevant information makes reasoning less effective and increases token costs.

Key takeaway: Store workflow state separately, retrieve memory selectively, and provide only relevant context to the model.

5. Orchestration: Workflow vs. Single Agent vs. Multi-Agent

Orchestration coordinates models, tools, memory, and execution steps.

There are three common approaches:

For example, a refund process usually follows a predictable workflow:

Classify → Retrieve → Validate → Approve → Execute → Respond

A predefined workflow may be more reliable than allowing an autonomous agent to plan everything.

Multi-agent systems introduce additional coordination, latency, and failure risks.

Each handoff between agents creates another opportunity for context loss or mistakes.

Key takeaway: Start with a simple workflow or single agent. Introduce multi-agent architecture only when it provides measurable benefits.

6. Evals, Observability, and Security: Production Controls

This is what separates a working demo from a production system.

Evals: Is the agent doing the right thing?

Traditional software uses deterministic tests.

AI agents introduce nondeterministic behavior. An agent can return valid JSON while selecting the wrong tool or applying the wrong refund policy.

Evaluation datasets should include:

• Normal requests.
• Ambiguous questions.
• Incorrect or missing information.
• Tool failures.
• Unauthorized requests.
• Prompt injection attempts.

Rerun evaluations whenever prompts, models, tools, or workflows change.

Observability: What actually happened?

When something goes wrong, we need to identify exactly which step failed.

Track:

• Model and prompt versions.
• Tool calls and results.
• Workflow execution traces.
• Latency and token costs.
• Errors, retries, and fallback decisions.

Tracing is especially important because one agent request may involve multiple LLM calls and external APIs.

Security: What if the agent makes a mistake?

Not all actions have the same risk.

Reading order information is different from issuing a refund.

For high-risk actions, follow:

LLM proposes → Backend validates → Human approves (if required) → Tool executes

Business rules and permissions must be enforced by application code, not just prompts.

Also treat customer messages, retrieved documents, and external tool responses as untrusted input to defend against prompt injection.

Key takeaway: Measure agent quality, trace execution, and restrict what the agent can do.

Putting It All Together

A production customer support agent might use this architecture:

These are possible implementation choices rather than requirements from the video.

Final Thoughts

The biggest lesson is that production AI agents are still software engineering systems.

LLMs introduce flexible reasoning, but traditional engineering principles remain essential:

• Reliability and fault tolerance.
• API and database design.
• State management.
• Testing and evaluation.
• Logging and observability.
• Security and authorization.

A successful AI agent combines probabilistic AI reasoning with deterministic software controls.

The goal is not to build the most autonomous or complicated agent. It is to build the simplest agent that reliably solves a real problem.

Reference: Agentic AI System Design was HARD until I Learned these 6 Concepts — YouTube

These are my learning notes based on the video, with a few additional engineering considerations and implementation examples.