Scaling laws for neural language models
An OpenAI paper showing that language-model error falls as a smooth power law in model size, data and compute, which turned building bigger models into a forecast labs could budget against.
What it was
Jared Kaplan, a Johns Hopkins physicist also working at OpenAI, and nine OpenAI colleagues, with Sam McCandlish second and Dario Amodei last, posted "Scaling Laws for Neural Language Models" to arXiv on January 23, 2020.
They trained many transformer language models of different sizes on different amounts of text and found that the test loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." Architectural choices such as depth and width mattered little by comparison. The paper also gave a rule for spending a fixed compute budget: train very large models on a relatively modest amount of data and stop "significantly before convergence."
What it changed
Four months later, on May 28, 2020, OpenAI posted GPT-3, a 175-billion-parameter model whose paper, with Kaplan and Amodei among its authors, reported performance rising with size across its tasks. The recipe was then revised by DeepMind. Jordan Hoffmann and 21 colleagues reported on March 29, 2022 that "current large language models are significantly undertrained," that model size and training data should grow in equal proportion, and that their 70-billion-parameter Chinchilla, trained on four times more data, beat the 280-billion-parameter Gopher and GPT-3.
Amodei has traced the idea to his speech-recognition work at Baidu. He told Lex Fridman in November 2024 that "a few people came to similar views around the same time," naming Rich Sutton's "bitter lesson" and Gwern's writing on the scaling hypothesis.
The arguments it moved
How close is human-level AI?
The paper gave short-timeline forecasts a curve to extrapolate. In June 2026 Amodei wrote that scaling laws "now have over a decade of empirical evidence behind them" and that "If these scaling laws continue for only a year or two longer, we are likely to get what I've called Powerful AI." The 2020 paper measured one thing, loss on predicting the next token of text; the forecast depends on lower loss continuing to bring broader abilities, and the paper itself did not test that.
Can language models reach general intelligence?
The paper made scale the central claim for language models and so gave their critics a target. Yann LeCun told MIT Technology Review in January 2026 that "people have had this illusion, or delusion, that it is a matter of time until we can scale them up to having human-level intelligence, and that is simply false." Amodei told Lex Fridman in November 2024 that doubters appear at each step: "at every stage of scaling, there are always arguments."
Positions it bears on
-
Dario Amodei, Timelines
He expects AI broadly better than humans at most cognitive work within a year or two of whenever he is asked, while insisting the estimate is a guess that could be wrong.
His forecasts of powerful AI within a year or two are extrapolations of the scaling laws this paper measured, as his June 2026 essay says.
-
Yann LeCun, Limits of large language models
Argues that autoregressive LLMs cannot reason or plan beyond their training data because they lack a model of the world, that scaling them will not produce human-level intelligence, and that the current paradigm will be replaced within a few years.
His argument that scaling language models will not produce human-level intelligence is the direct counter to the reading of this paper as a path to AGI.
-
Ilya Sutskever, Scaling and the return to research
Having championed scaling at OpenAI, he now says that simply multiplying compute and data will not transform what models can do, that pre-training on internet text will run out of data, and that progress depends again on new ideas tested on large computers.
Sutskever, whom Amodei credits with the remark that "these models just want to learn," now says the age of scaling that began in 2020 is over.
-
Gary Marcus, Limits of large language models
Argues that large language models are pattern-mimics without internal models of the world, that scaling them has reached diminishing returns, and that reliable AI needs hybrid systems adding symbolic reasoning and explicit knowledge.
Marcus argued in 2022 that scaling was approaching diminishing returns, the main published challenge to extrapolating these curves.
Sources
- Scaling Laws for Neural Language Models (arXiv, January 23, 2020)
- Language Models are Few-Shot Learners (arXiv, May 28, 2020)
- Training Compute-Optimal Large Language Models (arXiv, March 29, 2022)
- Dario Amodei on the Lex Fridman Podcast, transcript (November 2024)
- Policy on the AI Exponential (Dario Amodei, June 2026)
- Yann LeCun's new venture AMI Labs (MIT Technology Review, January 22, 2026)