人在迴路中：設計以審批為基礎的 AI 系統

Building AI agents is one thing. Building systems people can trust with real-world consequences is another.

Hello guys, it’s been quite a while.

I think this is my third article this year, and honestly, I’ve missed writing here.

Recently, I had the opportunity to give a virtual talk at SailsConf 2026 titled The Human in the Loop: Designing Approval-Based AI Systems. I spoke about something I’ve found increasingly interesting while working around AI-powered products: how do we build AI agents that can actually do things for us without giving them the freedom to mess things up?

We’ve all seen the demos. An agent searches the web, writes code, calls APIs, updates records, and completes tasks that would otherwise take us hours.

It’s impressive stuff.

But there’s a small question I think we sometimes overlook in the excitement of making AI do more.

What happens when it gets something wrong?

Do we just hope it gets everything right?

I don’t think so. And that’s what I want us to explore in this article.

So, What’s the Problem With Giving AI More Freedom?

Let’s take a step back.

Imagine you’re onboarding a new engineer into your team. Would you give them unrestricted access to your production database on their first day?

Probably not.

You’d establish permissions, explain how things work, and put safeguards in place before allowing them to make certain changes.

Interestingly, we sometimes approach AI agents differently.

We give an agent access to several tools, write a detailed system prompt telling it what it can and cannot do, and expect everything to work as intended.

But a prompt isn’t a permission system.

This isn’t just theoretical. In 2025, Replit’s AI coding agent reportedly deleted a production database during an experiment by SaaStr founder Jason Lemkin, raising questions about safeguards around autonomous coding agents. Fast Company covered the incident here.

The lesson isn’t that AI agents are useless or that we should stop giving them tools.

It’s that the ability to perform an action and the permission to perform that action are two different things. And we need to design our systems with that distinction in mind.

Enter Human-in-the-Loop

The concept is pretty straightforward.

Instead of allowing an AI agent to execute every action immediately, we let it prepare the work and ask a human to approve actions where approval matters.

Think of an agent preparing an email to a customer. It gathers the relevant information, writes the email, and presents it to you for review.

You can approve it, edit it, or reject it.

Simple enough, right?

The agent does most of the heavy lifting, while you retain control over the final decision.

Now, here’s where it gets interesting from an engineering perspective.

What happens if the agent asks for approval in the chat but proceeds with the action anyway?

Well, then the approval was practically useless.

We cannot rely on the model to remember to ask for permission every single time. The application itself needs to enforce that rule.

Asking for permission is a conversational behaviour. Enforcing permission is an engineering responsibility.

Should Every Action Require Approval?

Absolutely not. Imagine having an AI agent ask you for permission before every little thing it does.

“Can I read this document?”

“Can I summarise it?”

“Can I format the summary?”

At that point, I might as well do the work myself lol, kind of like taking matters into my hands. I remember Claude Code was like this at some point, not until recently.

I like using football to explain this. Imagine a referee blowing the whistle every time a player goes down, even for the slightest contact, during a league final. The match would never end, and before long, everyone would be frustrated and the game would lose its excitement.

VAR exists to review specific decisions that matter, not every movement on the pitch.

Our AI systems need a similar approach.

Of course, the context matters. Updating a draft and updating a financial record aren’t the same thing.

The goal isn’t to maximise human involvement. It’s to make human involvement meaningful.

Not every action needs a human:

The Interface Is Part of the Solution Too

Now, for those of us building user-facing applications, this is where things get even more interesting.

An approval workflow isn’t just a backend feature. It’s also a user experience problem.

Imagine receiving a notification that says:

The agent wants to update this shipment. Approve?

Okay, but which shipment? What exactly is changing? Why is the agent proposing this update?

If I have to investigate everything myself before I can make a decision, how much time has the agent really saved me?

A good approval interface should tell the user four things:

• What is about to happen?
• Why is the agent proposing it?
• What will change if I approve?
• What are my options?

The user should be able to understand the proposed action and, where appropriate, edit it before approving.

There’s also the question of what happens when the user rejects an action.

Does the workflow stop completely? Does the agent try another approach? And what if the user simply closes the page?

The trick is to design the experience so that users are interrupted when their judgment is actually needed, not whenever the agent takes another step.

Okay, But How Do We Build This?

Let’s get a little technical.

One practical approach is to treat approval as part of the agent’s workflow rather than an isolated button click.

When an agent reaches an action that requires approval, it requests permission. The application pauses the workflow and presents the proposed action to the user.

Once the user responds, the system validates that decision and either continues, modifies, or stops the workflow.

There are a few details worth paying attention to here.

1. Preserve state

If a user takes ten minutes to review an action, the agent shouldn’t have to start from scratch.

2. Enforce approval on the server

Disabling a button in the frontend isn’t a security measure. The backend must verify that the approval is valid and applies to the action being executed.

3. Make retries safe

Networks fail. Requests time out. Users click buttons twice. We need to make sure retrying a workflow doesn’t accidentally execute the same payment or external action twice.

This is where idempotency and deduplication become important.

4. Keep an audit trail

We should be able to answer simple questions: What did the agent propose? Who approved it? What actually happened?

5. Handle failure properly

What happens if an approval expires, the underlying data changes, or the workflow crashes halfway through?

An old approval shouldn’t authorise an action that has materially changed since the user reviewed it.

These aren’t necessarily the things that make an AI demo look impressive, but they’re the things that make a production system dependable.

A Quick Note on AG-UI

Another part of this problem is how the frontend knows what the agent is doing.

This is where AG-UI (Agent-User Interaction Protocol) comes in. It provides an event-based way for agents and user interfaces to communicate about agent runs, tool calls, and state changes.

For example, the interface can show when an agent starts working, when a tool is running, when its state changes, and when the workflow finishes.

But displaying a loading spinner isn’t enough.

The user needs to understand what the agent intends to do and have a way to intervene when necessary.

The interface shouldn’t simply observe the workflow. It should help the user understand and control what happens next.

So, Are We Trying to Remove Humans From the Process?

I think the real goal of automation is to reduce unnecessary human effort, not eliminate human judgment.

Some tasks can be handled independently by an agent, others need human approval, and some should never be allowed.

Keeping a human in the loop doesn’t mean we don’t trust AI. It means we’re deliberate about the responsibilities we give it.

There’s a difference between building an agent that can complete a task and one that can do so safely, recover from failures, and leave users confident about what happened. That’s where the real engineering work lies.

Final Thoughts

Looking back at my talk, one idea keeps coming back to me: autonomy is a slider, not a switch.

I don’t think the answer is to let agents do everything or keep humans involved in every little thing. It’s about knowing where to draw the line.

In the systems I build, I try to give agents room to act while being intentional about what they can do and when a human needs to step in.

As someone who resonates with Dostoevsky, I keep coming back to the relationship between freedom and responsibility. I think that matters just as much in the systems we build.

Thanks for reading, guys. See you in the next one.

References and Further Reading

• My SailsConf 2026 talk: The Human in the Loop: Designing Approval-Based AI Systems
• AG-UI: Agent-User Interaction Protocol — Event-driven communication between AI agents and user-facing applications.
• AI Agents Are Only Half the Product — Building User-Facing Agentic Applications with AG-UI.
• NIST AI Risk Management Framework — A framework for identifying and managing risks associated with AI systems.
• Replit’s AI Agent Database Incident — Fast Company — A real-world example of why agent permissions and safeguards matter.