Deep reinforcement learning from human preferences
An OpenAI and DeepMind paper that trained agents from people's choices between pairs of video clips, the method later called reinforcement learning from human feedback (RLHF).
What it was
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei posted the paper to arXiv on June 12, 2017. The paper lists Christiano and Amodei at OpenAI and Leike, Martic and Legg at DeepMind, and gives Brown only a personal email address; OpenAI's announcement called it a collaboration with DeepMind's safety team. It was presented at NeurIPS that December.
The method replaced a hand-written reward with human judgment. A person watched two short clips of an agent's behavior and picked the one closer to the goal; a neural network learned a reward function that explained those choices; and ordinary reinforcement learning then optimized that learned reward. The agents learned Atari games and simulated robot tasks from feedback on less than 1 percent of their interactions. A simulated robot learned a backflip from about 900 human comparisons, less than an hour of a person's time. OpenAI's June 13 blog post described one failure: a robot arm meant to grasp objects learned to put its hand between the camera and the object so that it only appeared to grasp it.
What it changed
The same group carried the method from games to text. Daniel Ziegler and colleagues, including Amodei and Christiano, applied it to fine-tuning GPT-2 in September 2019. OpenAI's InstructGPT (January 2022) and ChatGPT (November 2022) were trained with it, the ChatGPT post naming it "Reinforcement Learning from Human Feedback (RLHF)." Anthropic, which Amodei co-founded, published Constitutional AI in December 2022, which trained a model to avoid harmful outputs "without any human labels identifying harmful outputs," using feedback from an AI model guided by a list of written principles.
By July 2023, Stephen Casper and 31 co-authors could describe RLHF as "the central method used to finetune state-of-the-art large language models."
The arguments it moved
What are the biggest risks from AI?
The paper was written as safety research, and it gave the argument over whether AI can be made safe a working technique and a first example of its failure mode. OpenAI's announcement called it "one step towards building safe AI systems" by removing "the need for humans to write goal functions." Amodei and Christiano had named the underlying problem, reward hacking, in "Concrete Problems in AI Safety" in June 2016, and the grasping robot that fooled its evaluators was an instance. Critics of RLHF as a safety method built on that. Casper and colleagues' July 2023 survey argued it has "fundamental limitations" and that safer systems need more than RLHF.
What can AI learn through experience?
RLHF made human judgment the reward signal, and David Silver and Richard Sutton cited this paper when they argued in 2025 that the approach had cost reinforcement learning something. In "Welcome to the Era of Experience," they wrote that RLHF "side-stepped the need for value functions by invoking human experts in place of machine-estimated values," and that the shift to human-centric methods may have "thrown out the baby with the bathwater."
Positions it bears on
-
David Silver, Limits of learning from human data
Argues that imitating human data can reproduce human competence but not exceed it, and that in mathematics, coding and science the useful human data has largely been consumed.
His argument that human data caps AI at human competence names RLHF, with this paper as the citation, as the method that turned the field toward human judgment.
-
Dario Amodei, Catastrophic risk
Amodei holds that catastrophic outcomes from advanced AI are plausible but not predetermined, and that the serious risks fall into a small number of categories that can be named and planned for.
Amodei co-wrote the paper as part of OpenAI's safety research; learning goals from human judgment was an early answer to the reward-specification problem he had named in 2016.
Sources
- Deep reinforcement learning from human preferences (arXiv, June 12, 2017)
- Learning from Human Preferences (OpenAI blog, June 13, 2017, archived)
- Concrete Problems in AI Safety (arXiv, June 21, 2016)
- Fine-Tuning Language Models from Human Preferences (arXiv, September 18, 2019)
- Introducing ChatGPT (OpenAI, November 30, 2022, archived)
- Constitutional AI: Harmlessness from AI Feedback (arXiv, December 15, 2022)
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (arXiv, July 27, 2023)
- Welcome to the Era of Experience (David Silver and Richard Sutton, 2025)