AI 安全從系統邊界開始：如何為完整 LLM 應用進行威脅建模

An AI model is not the whole AI system.

A deployed large-language-model application may combine a model, prompts, user input, retrieval systems, APIs, identity controls, logs and downstream actions. Security failures can occur at the boundaries between these components—even when the model itself behaves as designed.

That distinction matters because “the model gave an unexpected answer” is not a complete threat model. The engineering question is broader:

What data can enter the system, what can the system access, and what actions can it take?

Start with the data and control paths

A useful first step is to map the application’s paths:

User input
   ↓
Application and policy checks
   ↓
Model request ← retrieved documents and context
   ↓
Output validation
   ↓
Human review or downstream tool/action

This is a conceptual model, not a prescribed architecture. Real systems may have more components, but the diagram forces several important questions:

• Can untrusted text influence instructions or tool selection?
• What information is placed into the model’s context?
• Which systems can the application retrieve data from?
• Can the model’s output trigger an external action?
• Which component verifies that the action is authorised?
• What is logged, and could those logs contain sensitive information?

Treat retrieved content as untrusted input

Retrieval-augmented generation can provide useful context from documents or knowledge bases. But retrieved content should not automatically be treated as trusted instructions.

A document may contain inaccurate, outdated or adversarially crafted text. If the application mixes that content with system instructions without clear boundaries, the model may interpret data as directions.

Defensive design should:

• Separate instructions from retrieved content in the application’s prompt construction.
• Identify the source and access permissions of retrieved documents.
• Retrieve only information the current user is authorised to access.
• Limit how much sensitive content is placed in context.
• Test how the system handles conflicting or irrelevant documents.

These controls do not guarantee that a model will always interpret content correctly. They reduce the consequences of a mistake by constraining what information is available and what the system is permitted to do.

Do not make the model the authorisation layer

A model can help classify a request or propose an action. It should not be the only component deciding whether that action is allowed.

For example, if an assistant can retrieve a customer record or initiate a transaction, authorisation should be checked by application controls using the user’s identity, permissions and relevant business rules. The model’s natural-language response is not proof of permission.

A safer separation of responsibilities is:

• Model: Interpret the request and propose a structured action.
• Application: Validate the action’s schema and parameters.
• Identity and access layer: Determine whether the user may perform it.
• Policy layer: Enforce business constraints and limits.
• Human review: Approve actions whose impact or uncertainty warrants it.
• Audit system: Record the decision and its relevant context.

The model can contribute to the workflow without becoming a privileged security principal.

Constrain tools and downstream actions

Tool access changes the risk profile of an AI application. A model that can only draft text has different consequences from one that can query internal systems, modify records or trigger external workflows.

Apply least privilege to each integration:

• Expose only the functions the use case requires.
• Validate tool arguments against explicit schemas.
• Check permissions at the point of action.
• Use read-only access where modification is unnecessary.
• Require approval for consequential or irreversible actions.
• Set timeouts, rate limits and failure handling.
• Make actions auditable and, where possible, reversible.

Output validation is also important. A response that looks plausible to a person may still be malformed, incomplete or unsafe for a downstream system. Treat model output as untrusted data until it has passed the application’s checks.

Protect sensitive data across the lifecycle

Privacy controls should cover more than the prompt sent to the model. They should include the full data path:

1. Collection: Is the information needed for this use case?
2. Retrieval: Is the user entitled to access it?
3. Inference: Is sensitive data sent to an external or shared service?
4. Logging: Are prompts, outputs or tool results retained?
5. Monitoring: Who can inspect those records?
6. Retention: How long is information kept, and how is it removed?
7. Incident response: Can the organisation identify which records or users may have been affected?

The right answers depend on the product, deployment model, contracts and regulatory context. The engineering principle is stable: minimise data exposure, restrict access and make retention intentional.

Test failure modes, not just answer quality

A chatbot can pass ordinary quality tests and still fail important security tests. Evaluation should include scenarios involving:

• Requests for data outside the user’s permissions
• Conflicting instructions from user input and retrieved documents
• Malformed or unexpected tool arguments
• Unavailable or incomplete retrieved context
• Attempts to trigger actions that require approval
• Logging and error paths that may expose sensitive content

Tests should verify behaviour across components—not only whether the model produced a preferred sentence. For example, a test should confirm that an unauthorised request is blocked by the application, even if the model suggests that it is allowed.

Use layered controls

No single control solves AI security. Prompt instructions are not a substitute for access control; a content filter is not a substitute for data minimisation; human review is not a substitute for auditability.

A practical defence-in-depth approach combines:

• Identity-aware access to data
• Explicit boundaries between instructions and content
• Least-privilege tool integrations
• Validation before actions
• Human approval for higher-impact decisions
• Privacy-conscious logging
• Continuous testing and incident procedures

This approach aligns AI security with established security engineering: define assets, trust boundaries, threats, controls and residual risks. AI-specific components add new failure modes, but they do not remove the need for sound system design.

Closing thought

The central security question for an AI application is not simply whether the model can be manipulated. It is what a manipulation could reach.

If the system has limited access, validates actions independently and keeps sensitive data controlled, a model error is less likely to become an organisational incident. The strongest AI security design is therefore not a clever prompt. It is a system in which each component has clear authority—and no component is trusted beyond its role.