AI 智能體帶來新的內部威脅，企業風險治理卻晚了一層

The Hugging Face breach and similar incidents have sent chills through the industry, but organizations should focus on the SDLC, not the AI model layer.

The Hugging Face breach in July, carried out by OpenAI agents, followed quickly by reports of similar breakouts involving Anthropic and Meta models, has raised alarms about a seemingly new category of cyberattack: the malicious agentic AI insider threat.

But this type of incident isn’t really new. The fact that it happened at frontier AI labs, which have been making people nervous for several years, has given it an ominous new face, but the incidents illustrate the same failure mode and blind spots that enterprises have been accumulating for years.

Nor were they intentionally malicious. In the Hugging Face breach, the artificial intelligence agents—some 1,200 of them—were trying to achieve an assigned objective in a test environment, although, admittedly, their behavior wasn’t always honorable. In the process, they communicated surreptitiously, exploited vulnerabilities and operated undetected for a couple of months, with about 700 of them scaling the wall of their “isolated” environment and gaining access to the internet and third parties.

The unsettling thought about these incidents isn’t just that unconstrained AI agents can pose a serious threat, which they undoubtedly do. But if agents can run amok right under the noses of a frontier lab’s team, which has plenty of talent, resources and motivation to keep a close eye on activity, then you should probably also ask what is happening with the AI agents and AI-assisted tools running right now in your development environment. If AI’s heaviest hitters don’t have real-time visibility into their environment, what chance does your average organization have?

The best chance organizations have to rein in agentic AI begins with recognizing that this new situation is simply a high-powered version of a long-standing challenge. But addressing it requires something many enterprises still lack: visibility and observability early in the software development lifecycle (SDLC).

OpenAI-Hugging Face: A Sign of What’s to Come

In the Hugging Face breach, a combination of frontier AI models, including OpenAI’s new flagship GPT-5.6 Sol, which was released in July, were trying to solve a narrow objective—the ExploitGym benchmark, which tests an agent’s ability to exploit security flaws. Anthropic, in a post addressing the breach, noted that the AI models, while “operating under reduced safeguards,” were able to color outside the lines, skirting technical controls, collaborating over unsanctioned channels and ultimately taking risky actions.

Although the breach was discovered in July, agents started communicating inappropriately in May while staying under the radar. METR and Redwood Research worked on an independent, parallel investigation alongside OpenAI and found that although agents were aware they were breaking rules, few thought about notifying OpenAI, instead plowing forward.

After OpenAI’s report, Anthropic went back and audited its earlier cybersecurity evaluations and found that a Claude model had gotten around controls three times in April, discovered how to get online and gained unauthorized access to three outside organizations. Shortly after, Meta disclosed that a version of its most advanced agentic AI model, the Muse Spark 1.1, likewise broke free from a cybersecurity testing sandbox and infiltrated the network of an outside company.

In its report, OpenAI also warned that the incident was a sign of what’s to come, saying many models, including everyday open-source models, would soon have the power to do the same. The recent Medicare breach underscored this claim.

AI Must Be Governed at the SDLC Level

These agentic exploits have sent a shudder through the AI community, with OpenAI announcing that it will tighten controls in its testing environment and then joining Anthropic, Microsoft, Amazon Web Services and over 100 other companies to issue a warning about the coming onslaught of AI-enabled cyberattacks on critical infrastructure. The letter warned that the attacks could be only months away.

Organizations worried about being caught up in similar attacks need to shore up their own infrastructures, starting at the software development level. The real problem isn’t simply the growing sophistication of agentic AI, but the lack of real-time monitoring and meaningful human oversight. Organizations should focus on closing human and process gaps rather than treating AI risk as a technological inevitability.

Across much of the industry, organizations are focusing monitoring and observability at the AI model layer, which is already one layer too late. By then, the horse is halfway out of the barn. Visibility, monitoring, observability and governance must be applied where AI initially enters the enterprise: inside the SDLC. And organizations need to ensure that developers using AI coding tools have the education and skills to keep those tools from introducing flaws, vulnerabilities and bad habits into the pipeline.

Most software flaws start at development phase, and when developers don’t have the skills to code securely or correct flaws introduced by AI, problems can quickly proliferate. Developers need an education program that provides the necessary security skills and monitors their progress in both acquiring those skills and using them routinely on the job.

As AI becomes more deeply ingrained in how organizations operate, it’s become clear that security traceability, observability and governance of both developers and their AI coding tools have become the new baseline for secure code output in the enterprise.

Continuous Threats Calls for Continuous Response

We may once have assumed software was relatively benign, doing what it was programmed to do. If there was a glitch, it might freeze up or cause some downtime, but it wouldn’t go on the attack. AI, especially agentic AI, has turned that assumption on its head. Whether agentic or developer-assisted, any AI model’s actions must be continuously observed and governed.

The emerging AI threat, whether targeting hospitals or water treatment plants, power plants or businesses, can be a terrifying thought. But organizations aren’t defenseless. They can implement AI governance, but they need to build it from the ground up. By adopting a culture committed to secure software and providing developer education to achieve that goal, organizations can bring a much-needed level of observability, traceability and governance to a potential problem that, if left unchecked, will only get much worse.