Scaling laws

Scaling laws are measured relationships showing that a model's error falls smoothly and predictably as it gets more parameters, more training data and more computing power. A 2020 OpenAI paper found these follow power laws over many orders of magnitude, and a 2022 DeepMind paper (Chinchilla) revised how to split a budget between model size and data. The scaling hypothesis is the further bet that capability keeps rising along these curves.

Why it comes up

Amodei, senior author of the 2020 paper, built his case for rapid progress on them; LeCun, Ng and Hassabis argue that more scale alone will not produce human-level intelligence.

People and milestones

Scaling laws for neural language models, 2020 GPT-3, 2020

Dario Amodei

Sam Altman

Ilya Sutskever

Gary Marcus

Sources

Scaling Laws for Neural Language Models (Kaplan and others, 2020)
Training Compute-Optimal Large Language Models (Hoffmann and others, 2022)