跳到正文
DEV Community· Kevin James·· 1 天前AI 評分42

AI Agent 能寫程式碼,但我們能放心讓它們在正式環境中運作嗎?

AI Agents Can Write Code. But Can We Trust Them in Production? 🤖

AI 導讀

AI Agent 已能修改程式碼、執行命令及調用 API,但具備執行能力不代表應獲準自主執行,尤其在正式環境中,操作失當可能令事故惡化。文章主張以政策與授權層分隔推理和執行,並為敏感操作加入人手審批、最小權限及審計記錄。正式環境的 Agent 亦須驗證操作結果,並記錄決策依據和執行情況。

正文

The AI Agent Revolution Is Exciting. But There's a Catch.

We're entering an interesting phase of software engineering.

AI tools are no longer limited to generating code snippets or answering technical questions.

Today, agents can:

  • Read and modify entire codebases.
  • Execute commands and interact with APIs.
  • Investigate production incidents.
  • Generate tests and automate development workflows.
  • Coordinate multiple tools to complete complex tasks.

That's genuinely exciting.

But there's a question I think deserves much more attention:

Just because an AI agent can perform an action, does that mean we should allow it to execute that action autonomously?

My answer is: Not always.

And I think this distinction will become increasingly important as more companies deploy AI agents into real production environments.

1. A Successful API Call Doesn't Mean a Successful Outcome

Imagine you're building an AI-powered operations assistant.

A user asks:

"Investigate why our order processing service is experiencing delays."

The agent discovers that a queue has accumulated thousands of messages.

It checks monitoring dashboards, reviews logs, and identifies a potentially problematic downstream service.

So far, everything looks great.

But then the agent decides to restart that service.

Technically, the restart API works.

The agent has the necessary permissions.

The operation completes successfully.

Unfortunately, the restart interrupts active transactions and makes the incident worse.

The problem wasn't that the agent lacked technical capability. The problem was that it lacked operational judgment and appropriate boundaries.

This is where agent engineering becomes more than prompt engineering.

We need systems that understand the difference between:

  • What an agent is technically capable of doing.
  • What an agent is authorized to do.
  • What an agent should do in the current situation.

Those three things are not always the same.

2. Agent Architecture Should Separate Reasoning From Execution

One design principle I strongly believe in is separating an agent's reasoning process from its ability to execute actions.

Consider a simplified architecture:

User Request
     |
     v
Agent / LLM
     |
     v
Planning & Tool Selection
     |
     v
Policy & Authorization Layer
     |
     +---- Read-Only Action
     |          |
     |          v
     |      Execute Tool
     |
     +---- Sensitive Action
                |
                v
         Human Approval
                |
                v
         Execute Tool
                |
                v
        Verify & Audit

The key idea is simple:

The LLM can propose an action, but the application must decide whether that action is allowed.

This distinction matters because language models are probabilistic systems.

They can misunderstand instructions, make incorrect assumptions, or be influenced by untrusted content.

Authorization and security enforcement shouldn't depend on the model remembering the rules.

They should be implemented in deterministic application code.

3. MCP Makes Integration Easier, Not Automatically Safer

The Model Context Protocol (MCP) has made it easier to connect AI agents with external tools and services.

An agent might discover tools such as:

get_order_status
search_customer_records
query_system_logs
restart_service
update_order_status

But tool availability shouldn't automatically imply permission to execute every operation.

For example, an agent investigating an order problem might need permission to read the order's status.

That doesn't mean it should also be allowed to cancel the order, issue a refund, or modify customer information.

A stronger design combines:

Least-privilege access: Give agents only the permissions necessary for the task.

Task-scoped authorization: Restrict operations to the specific resources involved in the user's request.

Human approval: Require explicit authorization for sensitive or irreversible actions.

Audit logging: Record what the agent requested, what was approved, and what actually happened.

Execution limits: Enforce timeouts, request budgets, rate limits, and bounded retries.

MCP provides a useful integration interface, but these safeguards still need to be designed and enforced by the surrounding system.

4. Reliability Is More Than Handling Exceptions

Another challenge with AI agents is that their failures aren't always obvious.

Traditional applications often fail in recognizable ways:

  • A request times out.
  • A database connection fails.
  • An API returns an error.
  • A deployment fails its health checks.

AI agents introduce another category of failure:

The system completes the workflow, but the result is wrong.

An agent might retrieve the wrong document, misunderstand a business rule, select an inappropriate tool, or report success before verifying the outcome.

That's why production agents need more than exception handling.

They need outcome verification.

For example, if an agent updates a record, the system should verify that the intended change occurred.

If an agent investigates an incident, its conclusions should be supported by actual evidence.

And if the agent doesn't have enough information, it should be able to say:

"I don't have sufficient evidence to complete this action safely."

Sometimes, declining to act is the correct engineering outcome.

5. Observability Should Explain Decisions, Not Just Errors

When an ordinary API fails, we investigate logs, metrics, and traces.

With AI agents, we need additional visibility.

I would want to understand:

  • What was the user's original objective?
  • Which tools did the agent select?
  • What evidence informed its decisions?
  • Which actions were approved or rejected?
  • How many retries occurred?
  • Did the final result satisfy the original objective?

A practical observability system might capture structured events like this:

{
  "workflow_id": "wf_123",
  "agent_action": "restart_service",
  "resource": "order-processing",
  "authorization": "approval_required",
  "approval_status": "rejected",
  "execution_status": "not_executed"
}

Of course, logs should avoid exposing credentials, sensitive customer data, or unnecessary model context.

The goal isn't simply to record everything.

The goal is to make agent behavior explainable, traceable, and accountable.

6. Human-in-the-Loop Doesn't Mean Human-in-Every-Step

One concern I often hear is that requiring human approval defeats the purpose of automation.

I don't think that's necessarily true.

Imagine an agent processing 5,000 documents.

Should a human approve every document retrieval?

Probably not.

Should a human approve every metadata lookup?

Again, probably not.

But should the agent request approval before deleting thousands of production records?

Absolutely.

The solution is risk-based autonomy.

Low-risk, read-only operations can often execute automatically.

Moderate-risk actions may require additional validation and tighter limits.

High-impact operations should require explicit authorization.

The challenge is finding the right balance between productivity and control.

Too many approval requests create friction.

Too few safeguards create operational risk.

The best agent systems should make routine actions easy while keeping consequential decisions visible.

7. The Future of AI Engineering Is Trustworthy Autonomy

I think the next major challenge in AI engineering won't simply be making models more capable.

It will be making AI systems more dependable.

Developers will need to think about:

Security: Can the agent access only what it needs?

Reliability: Can it recover safely when something goes wrong?

Observability: Can we understand why it made a decision?

Evaluation: Can we measure whether its outputs and actions are correct?

Human control: Can users approve, interrupt, or reverse consequential actions?

These are familiar software engineering principles, but agentic systems introduce new ways for them to fail.

And that's what makes this field so interesting.

We're not just building smarter assistants.

We're building software that can make decisions and take actions in environments where mistakes have real consequences.

That requires a different level of engineering discipline.

Final Thoughts

I'm genuinely excited about the direction AI agents are taking software development.

But I believe the most valuable agents won't necessarily be the ones that execute the most actions.

They'll be the ones that know when to act, when to verify, and when to ask for help.

A powerful agent can complete a task. A trustworthy agent can complete it without creating new problems.

And that's the standard I think we should be aiming for.


💬 I'd Love to Hear Your Thoughts!

I'm Andres, a software engineer interested in AI agents, backend systems, and modern software architecture.

I'm always interested in connecting with other developers, learning from different experiences, and exchanging ideas about building better software.

I'm particularly curious about three questions:

  1. Would you trust an AI agent to make changes directly in your production environment?
  2. What's the biggest challenge you've encountered while building or using AI agents?
  3. Do you think human approval should always be required for high-impact actions?

I'd love to hear different perspectives, especially from developers experimenting with agentic AI, MCP, and production automation.

Drop your thoughts in the comments. Let's have a meaningful engineering discussion! 🚀

來源:DEV Community · dev.to