InstructGPT
OpenAI fine-tuned GPT-3 on human demonstrations and rankings, and labelers preferred the 1.3-billion-parameter result to the 175-billion-parameter original; the method became the recipe for ChatGPT.
What it was
On January 27, 2022, OpenAI announced that InstructGPT models, "trained with humans in the loop," had become the default models on its API. The paper, "Training language models to follow instructions with human feedback," by Long Ouyang, Jeff Wu, Xu Jiang and 17 other OpenAI researchers including John Schulman, Paul Christiano, Jan Leike and Ryan Lowe, followed on arXiv on March 4, 2022.
The team hired about 40 contractors through Upwork and Scale AI. They wrote example answers to prompts, many of them submitted by API customers, and GPT-3 was fine-tuned on those answers. They then ranked several model outputs per prompt; a reward model trained on the rankings served as the reward for further training with reinforcement learning, the method of the 2017 paper by Christiano and colleagues. Labelers preferred outputs of the 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3, which has more than 100 times as many parameters.
What it changed
On November 30, 2022, OpenAI released ChatGPT, which its announcement called a "sibling model to InstructGPT" trained "using the same methods as InstructGPT, but with slight differences in the data collection setup." Two weeks later, on December 15, Anthropic published Constitutional AI, which trained a model to avoid harmful outputs "without any human labels identifying harmful outputs," with a written list of principles guiding AI-generated feedback instead.
The paper named a cost it called the "alignment tax": tuning for customer tasks made the model worse on some academic benchmarks, which OpenAI reduced by mixing pretraining data back into the reinforcement learning step.
The arguments it moved
Can language models reach general intelligence?
The paper's first sentence took a position in the scale argument: "Making language models bigger does not inherently make them better at following a user's intent." Its evidence was a model with one hundredth the parameters that the people rating it preferred, after fine-tuning alone. The resulting chat products then became the objects of the argument over understanding. Sam Altman's December 4, 2022 post "i am a stochastic parrot, and so r u" came four days after ChatGPT's release.
What are the biggest risks from AI?
InstructGPT made the question of whose preferences a model is aligned to an explicit part of the argument. OpenAI's January 2022 post said the model's behavior reflected "the preferences of our labelers, who directly produce the data used to train our models, and us researchers," and that, trained on English instructions, "it is biased towards the cultural values of English-speaking people." It also noted that a model trained to follow instructions "can be susceptible to misuse," showing InstructGPT writing a guide to breaking into a house. In July 2023, Stephen Casper and 31 co-authors surveyed what they called the "fundamental limitations" of RLHF, by then "the central method used to finetune state-of-the-art large language models."
Sources
- Training language models to follow instructions with human feedback (arXiv, March 4, 2022)
- Aligning Language Models to Follow Instructions (OpenAI, January 27, 2022, archived)
- Introducing ChatGPT (OpenAI, November 30, 2022, archived)
- Constitutional AI: Harmlessness from AI Feedback (arXiv, December 15, 2022)
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (arXiv, July 27, 2023)
- Sam Altman on X, December 4, 2022