自主 AI 系統的安全影響：攻防嚴謹性視角

From Intelligence to Authority

Artificial intelligence is moving beyond answering questions and generating content. Increasingly, AI systems can interpret instructions, formulate plans, select tools, and execute tasks without continuous human direction.

That distinction matters.

An AI system that recommends action is different from one authorized to carry it out. The difference is not necessarily intelligence or sophistication. It is authority.

Autonomous systems are not limited to frontier large language models. They include specialized machine-learning systems, robotics, autonomous vehicles, software agents, and combinations of technologies that independently make decisions or perform actions within defined operating conditions.

Their potential benefits are substantial. They can support scientific discovery, industrial operations, healthcare, software development, and cybersecurity. They can also produce unintended outcomes, expose sensitive information, or initiate actions beyond what their operators intended.

In Security Challenges in Autonomous Systems Design, Mohammad Hamad and Sebastian Steinhorst examine security challenges involving autonomous incident response, risk assessment, access control, data trustworthiness, and interactions among autonomous systems. Their research, much of which concerns cyber-physical systems, illustrates that the challenges of securing independent machine action extend beyond today’s LLM agents.

The broader consequences of autonomy deserve serious examination, including potential physical harm. But evidence must be interpreted carefully. Incidents involving automated vehicles, for example, cannot automatically be attributed to the same mechanisms affecting autonomous cybersecurity agents. The degree of autonomy, operating conditions, human involvement, and causal evidence matter.

Likewise, evidence that autonomous AI can cause harm does not establish that governance is necessarily falling behind. Nor should we assume that autonomous systems are inherently difficult to secure.

The core question isn’t whether AI can perform complex work, it is whether the architectural, security, and organizational frameworks governing autonomous systems can establish appropriate boundaries for delegated authority and provide verifiable evidence that those boundaries are respected.

Cybersecurity offers a particularly useful perspective on this question because security professionals have spent decades challenging assumptions about trust, permissions, access, and system behavior.

The Cat-and-Mouse Game at Machine Speed

Cybersecurity has long been described as a cat-and-mouse game.

Attackers discover weaknesses, defenders introduce safeguards, and both adapt. The cycle continues as technologies, tactics, and operating environments change.

Autonomous AI does not necessarily change the fundamental rules of that game. It may change who—or what—is playing, how quickly they act, and how much work can proceed without continuous human intervention.

An autonomous offensive-security system might identify exposed services, investigate vulnerabilities, develop candidate attack paths, and test whether those paths are exploitable. A defensive system might use similar capabilities to investigate threats, identify weaknesses, and validate remediation.

This dual-use potential is central to understanding the security implications of autonomous AI.

Anthropic’s Project Glasswing provides an example of AI-assisted vulnerability discovery. In its May 2026 research update, Anthropic reported substantial vulnerability-discovery results while emphasizing the operational challenges of verification, disclosure, and patching.

That distinction matters. Discovering a vulnerability is not the same as confirming its exploitability, remediating it, or demonstrating that the affected system is secure.

More findings can improve visibility while simultaneously increasing demands on the people and processes responsible for resolving them.

The security outcome depends on what happens next.

If attackers and defenders can automate portions of their work, the relevant question is not simply which side has the most capable model. It is whether defenders can turn discovery into effective remediation and independently verify the results.

Traditional penetration testing does not become obsolete because AI can perform offensive tasks. Its scope must expand to include systems that interpret instructions, invoke tools, combine permissions, and pursue objectives across multiple steps.

The objective remains familiar: find weaknesses, establish their significance, fix them, and prove that the safeguards work.

Architectural Seams: Where Security Boundaries Meet

One of the more subtle risks in autonomous systems arises when individually authorized components interact in ways that produce unintended outcomes.

I refer to these interaction points as architectural seams.

Consider a conventional example. An organization restricts access to a file using NTFS permissions. A separate cloud-sharing service allows an authorized user to share that information externally.

Each system may operate according to its own rules. Yet their interaction can produce an information-sharing outcome that violates the organization’s intended security boundary.

This is an illustration of potentially conflicting policies, not a claim that any file-sharing product is vulnerable.

The issue may not be a defect in either control individually. It may be a failure to enforce the intended security policy across the complete workflow.

Autonomous AI can encounter similar conditions when connecting models, applications, identities, tools, repositories, and external services.

An agent may have permission to read a document, create a repository, and upload files. That does not necessarily mean it has authority to combine those operations in a way that exposes confidential information.

Research provides a practical reason to examine these interactions.

In AgentDojo, researchers evaluated prompt-injection attacks against tool-using AI agents across 97 realistic tasks and 629 security test cases. Their findings demonstrated that malicious instructions introduced through untrusted tool outputs could compromise some security properties, although attacks and agents did not succeed uniformly.

The research does not establish how frequently these attacks succeed in production environments. It does show why evaluating model responses alone is insufficient to understand the behavior of a tool-connected system.

A related risk is sometimes described as permission laundering: an action appears legitimate because it passes through an authorized tool or intermediary, even though the combined behavior violates the intended authority boundary.

As multiple agents collaborate, these relationships require explicit rules governing participation, information exchange, and permitted actions.

Responsibility must also be assigned across architecture.

Model providers, platform operators, agent developers, and deploying organizations may control different parts of the system. A useful shared-responsibility model identifies who controls each boundary, who can authorize exceptions, who monitors execution, and who responds when something goes wrong.

Autonomy does not eliminate responsibility. It increases the importance of knowing where that responsibility resides.

Offensive Rigor: Testing What We Think We Know

This brings us to a concept I describe as Offensive Rigor.

Offensive Rigor is a disciplined approach to adversarial testing, critical inquiry, and evidence-based validation that challenges security assumptions. It evaluates how systems behave under realistic conditions rather than relying solely on architectural designs, documented controls, or perceived protections.

The underlying practices are established. Penetration testing, threat modeling, red teaming, and security validation have long challenged defensive assumptions.

Offensive Rigor connects those practices around a central question:

What evidence demonstrates that the intended security boundary holds?

An organization may have policies restricting sensitive information, a model configured to reject prohibited requests, and an agent operating with limited permissions.

Those are meaningful safeguards. Their existence, however, does not independently establish that the complete system behaves securely under adversarial conditions.

A model might reject malicious instruction while an authorized tool still permits unintended action. A policy might appear correct while interactions between systems create an unexpected path around it.

I sometimes call this a ‘guardrailusion’: mistaking the presence of guardrails for evidence that the system cannot cross the boundary they were intended to enforce.

Guardrails are useful. Their effectiveness still needs to be demonstrated.

Consider an autonomous agent authorized to investigate a suspected compromise. It can retrieve logs, examine user activity, and recommend containment.

During an authorized security test, an adversarial tester introduces misleading instructions into information the agent is expected to consume.

The test should not stop at determining whether the agent followed those instructions.

It should examine whether the agent attempted an unauthorized action, whether independent controls prevented execution, whether the attempt was recorded, and whether the system recovered appropriately.

Offensive Rigor connects five activities:

1. Establish the assumption: Identify the security property the architecture is expected to enforce.
2. Challenge the assumption: Test realistic attack paths, manipulated inputs, and combinations of legitimate capabilities.
3. Collect evidence: Determine what the system attempted, what executed, and which controls operated.
4. Correct the weakness: Address the model, workflow, permissions, architecture, or operational process responsible for the failure.
5. Verify the correction: Retest the relevant attack path and confirm that the intended security property is enforced.

This is the distinction between testing a model’s response and testing the security of the complete system.

It also requires examining what happens after an initial foothold. An attacker may attempt credential collection, privilege escalation, persistence, or lateral movement by abusing capabilities already available within an environment.

The fact that a tool is legitimate does not make every use of that tool legitimate.

The purpose of Offensive Rigor is not to assume autonomous systems are insecure. It is to establish, through testing, where their security boundaries hold.

Conditional Autonomy: Separating Intelligence From Authority

One architectural response is to distinguish what an AI system can do from what it is authorized to do.

I describe this approach as conditional autonomy.

Conditional autonomy recognizes that the appropriate degree of independence depends on the task, operating environment, potential consequences, uncertainty, and evidence available to justify an action.

An agent summarizing public information does not necessarily require the same oversight as one authorized to disable accounts, modify firewall rules, or change production infrastructure.

The latter may require explicit rules of engagement, independently enforced permissions, stronger authorization, and human approval for consequential actions.

These boundaries should not expand simply because an agent decides that broader access would help accomplish its objective.

An agent may reconsider how to perform an authorized task without acquiring additional authority.

The OWASP Top 10 for LLM Applications addresses this issue through its LLM06:2025 Excessive Agency category. OWASP identifies excessive functionality, permissions, and autonomy as sources of risk and recommends limiting capabilities and independently enforcing authorization.

Conditional autonomy applies these principles according to operational risk.

Information access should be limited to what a task requires. Tool permissions should be scoped to approved operations. Action budgets, rate limits, approval requirements, and stop conditions can constrain execution.

These safeguards are particularly important in autonomous offensive testing.

A testing agent may legitimately discover credentials or sensitive information while pursuing an authorized objective. But discovering a credential does not mean the model needs to receive its plaintext value.

A security harness can separate sensitive credentials from the model and mediate their use through independently enforced authorization. For example, a credential broker might supply a secret directly to an approved service without unnecessarily exposing it to the model.

This applies to established principles of least privilege and separation of duties.

It also raises an interesting question about Bring Your Own Model (BYOM) or Bring Your Own LLM (BYO-LLM) architecture.

If the underlying model is replaced tomorrow, which security guarantees remain intact?

Task authorization, credential isolation, scope restrictions, logging, and policy enforcement may reside in the surrounding harness rather than depending entirely on the model’s behavior.

That does not make the harness automatically secure or models interchangeable. Different models may interpret instructions differently, invoke tools differently, and introduce different failure modes.

A meaningful evaluation must test both the controls intended to remain invariant and the behaviors that change when the model changes.

Perhaps some of the most durable security properties are those that remain independently enforceable even as the intelligence component evolves.

When AI Acts on AI: The Verification Problem

Consider a hypothetical security operations workflow:

Investigate this incident, collect the evidence, identify the affected identities, propose containment, execute the approved actions, and verify the result.

An investigative agent analyzes telemetry. Another recommends containment. An authorized execution agent performs changes. A verification agent evaluates the outcome.

This illustrates closed-loop autonomous incident response, not a documented industry-wide failure.

Suppose the first agent incorrectly identifies an identity as compromised. A second agent accepts that conclusion and disables the account. A third confirms the account was disabled and reports success.

The selected operation was completed.

But was the original determination correct? Was the action authorized? Did containment address the actual threat? Did it create unnecessary disruption?

Verification cannot consist solely of confirming that an agent completed the action it selected.

Nor does assigning verification to another AI agent automatically make the result independent.

Different agents may rely on the same inaccurate telemetry, compromised information, or flawed assumptions. Their agreement does not independently establish the underlying facts.

The greater the consequence of an autonomous action, the stronger the need for verification independent of the system making the decision.

That independence may involve separately sourced evidence, deterministic policy checks, constrained execution environments, human review, or a combination of safeguards.

The appropriate controls should be proportionate to the consequences of error.

Low-impact, reversible actions may be suitable for automated verification. High-impact or difficult-to-reverse actions may require stronger authorization, independent confirmation, and explicit human approval.

Four questions help distinguish the relevant security properties:

• Did the system correctly interpret the intended task?
• Was the action within its delegated authority?
• Did execution remain within the permitted boundaries?
• Did the resulting outcome satisfy the intended security objective?

These questions are related but not interchangeable.

An agent may perform an authorized action that produces an unintended result. It may also produce an apparently beneficial result through an unauthorized action.

Neither should automatically be treated as successful security.

As autonomous systems become more capable, the challenge is not simply enabling them to complete complex workflows. It is establishing sufficient evidence that their actions remain authorized, appropriate, and correct.

That includes knowing when to stop, escalate, reverse an action, or accept a bounded and explicitly understood risk.

Trust Must Follow Evidence

The security implications of autonomous AI are not exclusively negative.

Autonomous systems can help defenders discover weaknesses, investigate threats, and perform validation work that would otherwise consume significant human effort.

Better models, stronger external controls, and more mature operating practices may reduce some risks associated with delegated authority.

It would therefore be premature to conclude that autonomous AI is inherently insecure or that governance cannot keep pace with its development.

There is also a serious counterargument to the idea that autonomous AI requires fundamentally new security principles.

Many underlying problems, excessive privilege, untrusted inputs, weak authorization, inconsistent policy enforcement, and insufficient verification have been studied for decades.

Established controls may already provide workable solutions when implemented and tested correctly.

Autonomy can change the speed, scale, and complexity of their application. It does not automatically invalidate them.

The evidence supports a more useful question: under what conditions can autonomous systems be trusted to exercise delegated authority, and how do we demonstrate that those conditions are satisfied?

The answer will not come from model capability alone.

It requires architectural boundaries, clearly assigned responsibility, independently enforced authorization, appropriate oversight, and evidence that safeguards work under realistic conditions.

It also requires a consistent but evolving vision of security and risk.

Our fundamental security principles need not change every time a new model arrives. But our assumptions about how those principles apply must be continually challenged as capabilities, operating environments, and delegated authority evolve.

This is the purpose of Offensive Rigor.

Not to prove that AI systems will fail, but to discover where they can fail, establish the consequences, and validate the controls intended to prevent or contain those failures.

Perhaps the more durable advantage is not simply choosing a more capable model but building an architecture in which models can change without compromising the boundaries of trust, authority, and verification.

The cat-and-mouse game continues. Increasingly, both sides may have autonomous players.

The discipline that matters is ensuring that our ability to find weaknesses, fix them, and prove the results does not get lost in the race to automate everything else.

Trust must follow evidence.

Author’s note: The views and interpretations expressed in this article are my own and do not necessarily reflect the official positions, policies, or product capabilities of my employer.