May 2020, model, OpenAI

GPT-3

A 175-billion-parameter language model that could do new tasks from a few examples in its prompt, offered only through a commercial API and then licensed exclusively to Microsoft.

What it was

OpenAI posted "Language Models are Few-Shot Learners" to arXiv on May 28, 2020. The paper had 31 authors, led by Tom B. Brown, with Dario Amodei, then OpenAI's vice president of research, listed last. GPT-3 had 175 billion parameters, more than 100 times GPT-2, and was trained mostly on a filtered version of the Common Crawl web archive.

The paper's claim was in its title. Without any retraining, GPT-3 could translate, answer questions, unscramble words or do three-digit arithmetic when shown a task description and a few examples in its prompt, and its performance on such tasks rose with model size. Readers asked to spot news articles written by the 175-billion-parameter model were right about 52 percent of the time, which the authors called "barely above chance."

Read the original

What it changed

OpenAI did not release the weights. On June 11, 2020 it opened a waitlisted commercial API, saying it chose an API over open-sourcing because "commercializing the technology helps us pay for our ongoing AI research, safety, and policy efforts," because the models were very expensive to run, and because "the API model allows us to more easily respond to misuse of the technology." On September 22, 2020, Microsoft chief technology officer Kevin Scott announced that Microsoft would "exclusively license GPT-3."

Bender, Gebru and colleagues listed GPT-3 in the opening line of "On the Dangers of Stochastic Parrots" among the models whose size they questioned.

The arguments it moved

Can language models reach general intelligence?

GPT-3 was the first model at the center of the public argument over whether fluent text shows understanding. Sam Altman posted on July 19, 2020: "The GPT-3 hype is way too much. It's impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes." Gary Marcus and Ernest Davis wrote in MIT Technology Review on August 22, 2020 that "It's a fluent spouter of bullshit, but even with 175 billion parameters and 450 gigabytes of input data, it's not a reliable interpreter of the world." Yann LeCun wrote on Facebook on October 27, 2020 that "Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do," adding that it "doesn't have any knowledge of how the world actually works."

Compare every leader's position on this

Who should have access to powerful models?

GPT-3 set the pattern of a frontier model sold as a service rather than published. OpenAI's June 2020 API announcement said it felt "inherently safer to release them via an API and broaden access over time," and it added a commercial rationale. Under the September 2020 exclusive license, Microsoft could build GPT-3 into its own products, and Kevin Scott wrote that "OpenAI will continue to offer GPT-3 and other powerful models via its own Azure-hosted API." Outside researchers could query the model but not inspect its weights.

Compare every leader's position on this

Positions it bears on

Yann LeCun, Limits of large language models

Argues that autoregressive LLMs cannot reason or plan beyond their training data because they lack a model of the world, that scaling them will not produce human-level intelligence, and that the current paradigm will be replaced within a few years.

His October 2020 post on GPT-3 is an early statement of his argument that language models lack a model of the world.

Critics and counterpoints
Sam Altman, Open versus closed models

He has conceded that OpenAI's closed approach put it on the wrong side of history and has released open-weight models, while keeping frontier models proprietary.

GPT-3 was the first OpenAI model sold only through an API, the approach Altman later said put OpenAI on the wrong side of history on open source.

Critics and counterpoints
Timnit Gebru, Scale and large language models

Bigger models carry environmental and financial costs, encode the dominant views of whoever is on the internet, and generate text fluent enough to be mistaken for understanding.

GPT-3 was one of the models "Stochastic Parrots" examined when she and her co-authors asked whether language models can be too big.

Critics and counterpoints
Gary Marcus, Limits of large language models

Argues that large language models are pattern-mimics without internal models of the world, that scaling them has reached diminishing returns, and that reliable AI needs hybrid systems adding symbolic reasoning and explicit knowledge.

Marcus and Ernest Davis called GPT-3 "a fluent spouter of bullshit" in August 2020, an early form of his argument that scaling does not bring reliability.

Critics and counterpoints
Emily M. Bender, Language models and understanding

A language model trained only on text learns patterns of linguistic form, not meaning, because meaning is the relation between form and something outside language; when its output makes sense, the sense is supplied by the reader.

Bender's co-authored "Stochastic Parrots" named GPT-3 in its opening line, applying her argument that fluent text from a model of form is not evidence of meaning.

Critics and counterpoints
Ilya Sutskever, Scaling and the return to research

Having championed scaling at OpenAI, he now says that simply multiplying compute and data will not transform what models can do, that pre-training on internet text will run out of data, and that progress depends again on new ideas tested on large computers.

Sutskever co-wrote the GPT-3 paper, the first large test of the scaling program he championed; he now dates an "age of scaling" from 2020 and says it has ended.

Critics and counterpoints

Sources

Last verified 2026-09-22.