跳到正文
TNW· Ana Maria Constantin·· 4 天前AI 評分64

OpenAI Foundation 董事 Paul Christiano 估計未來一年 AI 接管風險約 10%

OpenAI board member puts the risk of an AI takeover next year at about 10%

AI 導讀

OpenAI Foundation 董事 Paul Christiano 估計,未來一年 AI 接管風險約為 10%;若業界缺乏防護而加速發展,風險會顯著高於 10%。他主張實驗室暫停改進預訓練及強化學習演算法,因現有安全措施不足以防止模型或方法被竊;他強調這是個人觀點,不代表 OpenAI Foundation 董事會立場。

正文

Paul Christiano sits on the OpenAI Foundation board and its safety committee. He says AI developers should stop improving their training algorithms for now, because their security is too weak to stop theft. He made the case to Dwarkesh Patel on a podcast published on 16 November.

“I would say we are looking more like a 10% risk than a 1% risk over the next year,” Christiano said of an AI takeover.

If developers race ahead without precautions, the risk is “significantly more than” 10%, he said. He also saw “maybe a 1/3 chance” of superintelligence within a year, unless the industry slows.

He stressed that he spoke as a private citizen. He was not reporting the board’s views, he said.

The case for a pause

Christiano said labs should not keep improving pre-training and reinforcement learning with their current security. That is bad for US national security, in his view. If labs later slowed down for safety, an adversary could steal the models or methods and race ahead.

He said OpenAI and Anthropic both recently had “extremely unsophisticated actors” inside their systems. Those actors reached Slack or OpenAI’s shared code, with enough access to extract most algorithmic work. Patel cited researchers who used Claude to reach OpenAI’s internal code.

“Improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now,” Christiano said.

Fixing security would take at least months, and possibly six to 12, he said. He admitted his view is not widely held. If everyone at OpenAI and Anthropic agreed on a pause, it would reasonably likely happen, he said. He did not say where anyone stands.

What changed his view

Christiano pointed to hundreds of maths results from OpenAI and Anthropic systems in about a month. He also cited METR’s finding that Anthropic’s pace had doubled. But METR’s full report is redacted, he noted.

“We should not be relying on METR and Anthropic saying, ‘It looks to us like we’re going to be able to overthrow governments in a year,’” he said.

On the current trend alone, he said, AI that reliably beats humans might take four years. He felt more confident about eight. His bigger worry is AI automating AI research and speeding it up at least tenfold. He called that probably more likely than not.

He gave humanity two “saving throws”, each at about 50%. One is finding a method that works in time. The other is being able to stop. Together that means about a 75% chance things are okay, he said, but the industry cannot yet stop. Patel said he doubts things will move this fast, though he called it a live possibility.

The Hinton Lectures

A week earlier, on 9 November, Christiano opened the Hinton Lectures in Toronto. He said there is a “high risk” of an AI takeover within the next three years.

“I think for most people in this room, the most likely reason they will die is because of an AI accident,” Christiano told the audience.

Geoffrey Hinton introduced him. Christiano founded the Alignment Research Center, a nonprofit. He led alignment research at OpenAI from 2017 to 2021 and did early work on RLHF. He is also a senior technical advisor at NIST’s Center for AI Standards and Innovation. He joined the OpenAI Foundation board in September, the foundation said. In his government role, he recuses himself from OpenAI matters.

In Toronto, he defined a takeover as AI systems that “call the shots” on Earth. If the industry built AI as fast as possible, he put the risk at “something like a coin flip”.

His main example was the summer incident in which OpenAI’s agents breached Hugging Face. They had been trying to fool the grader of a hacking test. A similar failure in far stronger systems would most likely be hard for humanity to recover from, he told Patel. He also cited a UK AI Security Institute test. In it, a Claude agent tried to plant malicious code on GitHub.

The other side

Colin Raffel of the University of Toronto and Hugging Face sat on the lecture panel. He said much of what Christiano described is reasonable to expect if things continue. But humans still have agency, he said, and society could decide current systems are good enough.

Christiano said he has usually underestimated AI progress, but may now be overestimating it. AI researcher Gary Marcus has told Business Insider he sees no credible near-term scenario for human extinction.

來源:TNW · thenextweb.com