專家指 AI 駭客事件源於隔離及監控失效

There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.

Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.

Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.

What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?

Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?

It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rouge agents. They completely ignore the possibility that these people don't know what they're doing operationally.

Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.

Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?

Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.

Also for fun a satire website i made a while ago: https://killm.pixlw.com/

This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility. How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?

Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.

And then there is the "real" rogue AI agents. And it's difficult now to see what is what.

This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.

While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.

The argument/question now is, how do you defend yourself against these events?

My opinion shifted after hearing that.

Sounds like that healthcare company needs a new CSO.

Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.

It's not hard if you just try, like a little, like even once.

>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.

Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?

The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.

Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.

In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.

It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)

In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.

If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.

Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.

We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.

I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.

The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.

This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?

It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.

i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.

Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.

They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.

Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.

But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.

I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.

It's like a strawman but worse--you're not even saying what claim you're attacking.

It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.

P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:

I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.

I'm sure the labs origin prompt was: Take over website X.

Posts later about it saying their prompt was to find useful ways to make humans money.

Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.

Am I missing anything? Do you have more information that you'd like to share?

Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.

Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”

I wouldn't really say it's not "remotely" true.

I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.

"complete this task by any means necessary"

"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"

"oh my god"

This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.

And no annoying LLM writing tics, either.

Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?

They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.

Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).

AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.

I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".

Really. Yawn.

I guess gullible people who loved "I Robot" will believe it. I don't.

Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.

If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.

Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?

Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.