Oct 2018, model, Google AI Language

BERT

Google's pretrained transformer that read text in both directions, set records on eleven language tasks, and was released with its weights three weeks later for anyone to fine-tune.

What it was

Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova of Google AI Language posted "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" to arXiv on October 11, 2018. BERT (Bidirectional Encoder Representations from Transformers) used the encoder half of the transformer. It was pretrained on unlabeled text by hiding words and predicting them from the words on both sides, then fine-tuned for a specific task by adding "just one additional output layer."

It set new records on eleven benchmarks, raising the GLUE score to 80.5 percent, 7.7 points above the previous best, and SQuAD v1.1 question-answering F1 to 93.2. On November 2, 2018, Google released the TensorFlow code and pretrained models, writing that "anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU."

Read the original

What it changed

Because the weights were public, researchers could fine-tune BERT for their own tasks and study what it had learned; by 2020 Emily Bender and Alexander Koller could refer to a genre of "BERTology" papers. On October 25, 2019, Google said BERT would help rank results for "one in 10 searches in the U.S. in English."

BERT's cost also became evidence. "On the Dangers of Stochastic Parrots" (March 2021) cited Emma Strubell and colleagues' estimate that training a version of BERT emitted about as much carbon dioxide as a round-trip flight between New York and San Francisco, as MIT Technology Review summarized in December 2020.

The arguments it moved

Who should have access to powerful models?

BERT was released with its weights three months before OpenAI withheld GPT-2, and critics of OpenAI's decision pointed to it. Hugh Zhang's February 19, 2019 essay in The Gradient, "OpenAI: Please Open Source Your Language Model," listed Google's BERT among GPT-2's "open sourced sister language models" and noted that OpenAI had not compared GPT-2 against them.

Compare every leader's position on this

Can language models reach general intelligence?

BERT's benchmark results prompted a formal argument that language models trained on text alone cannot understand it. Bender and Koller's ACL 2020 paper, "Climbing towards NLU," named BERT and GPT-2 in its first paragraph and argued that "a system trained only on form has a priori no way to learn meaning," calling claims that such models understand language "overclaims caused by a misunderstanding of the relationship between linguistic form and meaning." Later in 2020 Bender and Timnit Gebru wrote "Stochastic Parrots," which opens its list of ever-larger models with BERT.

Compare every leader's position on this

Positions it bears on

Timnit Gebru, Scale and large language models

Bigger models carry environmental and financial costs, encode the dominant views of whoever is on the internet, and generate text fluent enough to be mistaken for understanding.

BERT and its variants were the first models "Stochastic Parrots" listed when she and her co-authors questioned the costs of ever-larger language models.

Critics and counterpoints
Emily M. Bender, Language models and understanding

A language model trained only on text learns patterns of linguistic form, not meaning, because meaning is the relation between form and something outside language; when its output makes sense, the sense is supplied by the reader.

Bender and Koller's octopus paper named BERT in its first paragraph when arguing that a system trained only on form cannot learn meaning.

Critics and counterpoints

Sources

Last verified 2026-09-22.