OpenAI 與 DeepMind 安全團隊合作開發從人類偏好推斷意願的演算法
Learning from human preferences
OpenAI 與 DeepMind 安全團隊合作開發了一種演算法,透過人類判斷兩種候選行為何者較好,推斷人類想要甚麼。他們將減少人類撰寫目標函數的需要視為構建安全 AI 系統的一步,因以簡單代理目標代表複雜目標,或把目標設定得稍有偏差,都可能導致不良甚至危險行為。
材料呈現以人類比較兩種候選行為的回饋推斷其意願的演算法思路,並連結到降低複雜目標函數設定偏差的安全目標。
Learning from human preferences
Read paper(opens in a new window)

Loading…
Share
One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.
We present a learning algorithm that uses small amounts of human feedback to solve modern RL environments. Machine learning systems with human feedbackhave(opens in a new window)been(opens in a new window)explored(opens in a new window)before(opens in a new window), but we’ve scaled up the approach to be able to work on much more complicated tasks. Our algorithm needed 900 bits of feedback from a human evaluator to learn to backflip—a seemingly simple task which is simple to judge butchallengingto specify.

Our algorithm learned to backflip using around 900 individual bits of feedback from the human evaluator.
The overall training process is a 3-step feedback cycle between the human, the agent’s understanding of the goal, and the RL training.

Our AI agent starts by acting randomly in the environment. Periodically, two video clips of its behavior are given to a human, and the human decides which of the two clips is closest to fulfilling its goal—in this case, a backflip. The AI gradually builds a model of the goal of the task by finding the reward function that best explains the human’s judgments. It then uses RL to learn how to achieve that goal. As its behavior improves, it continues to ask for human feedback on trajectory pairs where it’s most uncertain about which is better, and further refines its understanding of the goal.
Our approach demonstrates promising sample efficiency—as stated previously, the backflip video required under 1000 bits of human feedback. It took less than an hour of a human evaluator’s time, while in the background the policy accumulated about 70 hours of overall experience (simulated at a much faster rate than real-time.) We will continue to work on reducing the amount of feedback a human needs to supply. You can see a sped-up version of the training process in the following video.
We’ve tested our method on a number of tasks in the simulated robotics and Atari domains (without being given access to the reward function: so in Atari, without having access to the game score). Our agents can learn from human feedback to achieve strong and sometimes superhuman performance in many of the environments we tested. In the following animation you can see agents trained with our technique playing a variety of Atari games. The horizontal bar on the right hand side of each frame represent’s each agents prediction about how much a human evaluator would approve of their current behavior. These visualizations indicate that agents trained with human feedback learn to value oxygen in Seaquest (left), anticipate rewards in Breakout and Pong (center), or work out how to recover from crashes in Enduro(right).




Note there’s no need for the feedback to align with the environment’s normal reward function: we can, for example, train our agents to precisely keep even with other cars in Enduro rather than maximizing game score by passing them. We also sometimes find that learning from feedback does better than reinforcement learning with the normal reward function, because the human shapes the reward better than whoever wrote the environment’s reward.
Challenges
Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors look correct, so if the human doesn’t have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it, as shown below.

We addressed this particular problem by adding in visual cues (the thick white lines in the above animation) to make it easy for the human evaluators to estimate depth.
The research described in this post was done in collaboration with Jan Leike, Miljan Martic, and Shane Legg at DeepMind. Our two organizations plan to continue to collaborate on topics that touch on long-term AI safety. We think that techniques like this are a step towards safe AI systems capable of learning human-centric goals, and can complement and extend existing approaches like reinforcement and imitation learning. This post is representative of the work done by OpenAI’s safety team; if you’re interested in working on problems like this, pleasejoin us!
Footnote


By comparison, we took two hours to write our own reward function (the animation in the above right) to get a robot to backflip, and though it succeeds it’s a lot less elegant than the one trained simply through human feedback (top left). We think there are many cases where human feedback could let us specify a specific goal more intuitively and quickly than is possible by manually hand-crafting the objective.
You can replicate this backflip ingym(opens in a new window)with the following reward function for Hopper:
Python
1def reward_fn(a, ob):2 backroll = -ob[7]3 height = ob[0]4 vel_act = a[0] * ob[8] + a[1] * ob[9] + a[2] * ob[10]5 backslide = -ob[5]6 return backroll * (1.0 + .3 * height + .1 * vel_act + .05 * backslide)
Authors
Dario Amodei, Paul Christiano, Alex Ray
來源:OpenAI News · openai.com