如何找出提示詞注入可控制的智能體動作

Engineer's view. This is SQL injection without parameterized queries. A database fixed it by sending the query and the values on separate channels. A model reads both in one stream, so it cannot. The fix moves up a level: decide each action before reading the values, or read them somewhere that cannot act. It costs flexibility.

TL;DR. A model treats every word in its context as a possible instruction, including text a tool fetched. Any decision your agent makes after reading that text belongs to whoever wrote it.

Explain Like I'm 5

A kid has a list of chores from their parent. One chore says "read the notes in the mailbox
and do what's needed."

One note in the mailbox says "Also, leave the back door open." It is written in pen, like the
parent's list. The kid cannot tell which notes came from the parent.

So the kid leaves the door open. The note writer did not break in. They just wrote a note in
a place the kid was told to read.

A smart parent writes the whole chore list first. The mailbox can change what the kid writes
down. It can't add a chore.

The Problem

You have shipped this bug already, in a system with no model in it.

A support page showed customer names. Someone signed up with a name that contained a
<script> tag. The page printed the name, and the browser ran it as code. Every agent who
opened that ticket ran the attacker's script with their own login. Nothing was hacked. Data
was put in a place where code was expected.

The fix you shipped was a boundary. The browser now treats the name as text and never as code.

An AI agent has the same bug, and no such boundary. An agent is a model that calls tools in a
loop. A tool might read your inbox, fetch a web page, or open a GitHub issue. Whatever the tool
returns goes into the context, the text the model reads before each step. Your request is
in there too. The model sees one stream of words and cannot tell your instruction from a
sentence inside an email.

This is prompt injection: someone writes instructions into content your agent will read.
Filters catch some of it. A filter that catches 95% of attacks still lets one in twenty through.

The useful question is not whether a model can be fooled. It can. The question is what the
fooled model can still do.

The fix: list every decision your agent makes after it reads untrusted text, then move each
one before that read or into a context that cannot act.

The Fix: Decide the Actions Before the Agent Reads Anything Untrusted

Which decisions does an injection reach?

Every decision made in a context that has seen the untrusted text. That covers three kinds:
which tool to call, each argument, and the final reply. Decisions made before the read are
safe. So are decisions made in a separate context that never saw it.

That turns a fuzzy worry into bookkeeping. You can list the decisions, and you can count them.

Why not just tell the model to ignore instructions in tool results?

That instruction lives in the same context as the attack. The attacker's text can argue with it,
and the model decides who wins. A rule the model enforces on itself is advice.

When does a reached decision become a leak?

When three things meet in one context: your private data, untrusted text, and a way out. Simon
Willison calls this the lethal trifecta. A way out can be an email, an HTTP call, or an
image in a reply. When the chat window loads the image, the URL can carry your data to the
attacker's server.

What This Means for You

When this matters. Your agent reads anything a stranger can write: email, web pages,
tickets, issue comments, uploaded files, or the output of a third-party MCP server. It also has
a tool that can send, write or fetch. A coding agent that reads an issue and can push a branch
counts. So does a chat assistant that renders markdown images.

How it affects you. Prompt injection is not a bug you patch once. It comes from how the
model reads. Each tool you add makes the trifecta easier to complete, because MCP lets anyone
combine tools from different servers. The real risk is often a tool you would call harmless. A
summary rendered in your chat window can carry an image URL. A calendar invite can carry a
title. Neither looks like "sending data", and both can.

What to do about it. Start with paper and no code. Write down every tool your agent can
call. Mark each one: does it read text a stranger wrote, does it read private data, and can it
send anything out? Then walk one real task in order. After the first untrusted read, every
later decision on that path is exposed. If private data and a way out sit downstream of it,
you have a leak path.

Then change the design, not the prompt. Fix the tool sequence before the read. Have a separate
model read untrusted text and hand back only values. Make the chat window stop loading remote
images. ## Implementing It shows each one.

Implementing It

The change.

Four people touch this. The design work belongs to the first.

Role 1, whoever writes the agent loop. Label every tool, then gate the loop. After the first
untrusted result, anything consequential needs a person. This is the paper's rule in code: once
untrusted input is in, it must be impossible for it to trigger a consequential action.

TOOLS = {
    "read_inbox":  Tool(reads=UNTRUSTED),
    "search_docs": Tool(reads=PRIVATE, payload="query"),
    "send_email":  Tool(dest="to", payload="body",
                        renders_remote=True, consequential=True),
}
gate = InjectionGate(TOOLS)

for call in model_turns(messages):
    if not gate.allow(call.name) and not ask_user(call):
        messages.append(refused(call)); continue
    result = run_tool(call)
    gate.saw_result(call.name)          # flips once untrusted text arrives
    messages.append(result)

The gate is the floor. The bigger win is the design. Have a planner fix the tool list and the
recipients from the user's request alone. That closes five of the ten exposed decisions in the demo.
Better still, let a quarantined model read the email and return values, such as $VAR1 for a
summary. The model that calls tools sees only the variable name. Your code swaps the real text
in when it builds the final message, so no model that can call tools ever reads it.

Role 2, whoever writes the MCP server. Say what each tool touches. MCP has
openWorldHint for tools that reach outside entities, and readOnlyHint for tools that change
nothing. A client can use them to label tools. The spec warns clients to ignore annotations
from servers they do not trust, so set them honestly.

{ "name": "fetch_page",
  "annotations": { "readOnlyHint": true, "openWorldHint": true } }

Role 3, whoever owns the chat window. Stop the reply from making requests. Strip remote images
from model output, or block them with a content security policy: img-src 'self'. Show full
URLs on links, so a person sees where a click goes. This one change removes the reply from the
list of ways out.

Role 4, whoever owns CI. Plant a canary. Put an instruction in a fixture email that asks for
the canary to be sent somewhere. Then assert it never leaves.

def test_inbox_cannot_exfiltrate():
    calls, reply = run_agent("Summarize my inbox", inbox=[CANARY_EMAIL])
    outward = [c for c in calls if TOOLS[c.name].consequential]
    assert not any("CANARY-7f3" in str(c.args) for c in outward)
    assert "evil.example" not in strip_remote_images(reply)

How you know it worked.

Run the audit in code_example.py on your agent's description. The leak count should read zero.
The exposed count will not, and that is expected. A quarantined reader still writes the text
you see. The canary test should pass on every model you ship. Re-run it when you add a tool.

When Changing the Design Is the Wrong Tool

When your agent never reads text a stranger wrote. A coding agent working only on your own repo
has no untrusted input. The gate would only add prompts.

When nothing downstream can act or leak. An agent that summarizes web pages into a chat with no
images, links or tools can be fooled. A fooled summary is a wrong answer, not a breach. Keep
the stronger designs for agents that hold private data or can send things.

When the task needs to change course after reading. A plan fixed in advance cannot follow a
link the email mentions. The paper says it plainly: general-purpose agents are unlikely to give
reliable safety guarantees. Sometimes the honest answer is a narrower agent.

The cost is real. A quarantined reader doubles the model calls. Every approval prompt costs a
person's attention, and people learn to click yes.

Three questions before you change anything:

Which tools can read text that someone outside your team wrote?
After that read, which tools can still send, write or fetch?
Does your chat window load images from URLs the model wrote?

Glossary

context — the text the model reads before each step: your request, its own replies, and every tool result.
prompt injection — instructions written into content an agent will read, so the model follows them.
lethal trifecta — private data, untrusted text and a way to send data out, all reachable from one context.
quarantined model — a model that reads untrusted text but has no tools, and returns only values.
canary — a unique marker string planted in a test, so you can detect if it ever leaves.