The transformer
Eight Google-affiliated researchers replaced recurrence with attention in "Attention Is All You Need," and the architecture became the basis of GPT, BERT and nearly every large language model after them.
What it was
"Attention Is All You Need" was posted to arXiv on June 12, 2017 by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, listed as equal contributors. Vaswani, Shazeer and Kaiser were at Google Brain; Parmar, Uszkoreit and Jones at Google Research; Gomez, an intern, at the University of Toronto; and Polosukhin, who had left Google earlier that year, gave a personal address. It was presented at NeurIPS in December 2017.
Earlier translation systems read a sentence one word at a time with recurrent neural networks. The transformer dropped recurrence and let every word attend directly to every other word, which meant a whole sequence could be processed in parallel on GPUs. It scored 28.4 BLEU on WMT 2014 English-to-German, more than 2 points above the previous best, and set a single-model record of 41.0 on English-to-French "after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature."
What it changed
Google published the architecture openly, and others built products on it first. Wired's Steven Levy reported in March 2024 that soon after publication OpenAI's chief scientist Ilya Sutskever suggested that Alec Radford work on the idea, which led to the first GPT models; Google put transformers into its translation tool in 2018 and released BERT that October. The transformer underlies GPT-2 (2019), GPT-3 (2020), InstructGPT and ChatGPT (2022) and the models that followed.
By March 2024 all eight authors had left Google. Seven had founded companies, among them Character.AI, Cohere, Sakana AI, Inceptive, Adept and NEAR, and Kaiser had joined OpenAI.
The arguments it moved
Can language models reach general intelligence?
The argument over whether language models understand, and whether scaling them leads to general intelligence, is an argument about transformers trained to predict text. Their parallelism made the scaling experiments of 2019 and 2020 affordable, and Kaplan and colleagues' January 2020 scaling-laws paper was measured on transformers. Yann LeCun has argued since at least 2022 that autoregressive language models "can't truly reason or plan, because they lack a model of the world," as he told MIT Technology Review in January 2026.
Who should have access to powerful models?
Google invented the transformer and published it, and OpenAI turned it into products first. Sam Altman told Wired's Steven Levy, "When the transformer paper came out, I don't think anyone at Google realized what it meant." Uszkoreit disputed that, saying "It was pretty evident to us that transformers could do really magical things." Asked why Google had not launched a chatbot first, Sundar Pichai said, "The fact is, we can do more after people had seen how it works."
Positions it bears on
-
Yann LeCun, Limits of large language models
Argues that autoregressive LLMs cannot reason or plan beyond their training data because they lack a model of the world, that scaling them will not produce human-level intelligence, and that the current paradigm will be replaced within a few years.
His critique is aimed at autoregressive transformer language models, the use of this architecture that became dominant after 2018.
