Transformers & Attention

In 2017, eight Google researchers wrote a paper with a Beatles-inspired title. It now has over 173,000 citations, making it one of the ten most-cited papers of the 21st century. Every single one of those researchers left Google to start or join AI companies. The paper introduced an architecture called the Transformer -- and it's the reason you can talk to ChatGPT, Claude, Gemini, or any other large language model today.

The Problem with RNNs

Before transformers, the dominant architecture for processing language was the Recurrent Neural Network (RNN)A neural network that processes sequences one element at a time, passing hidden state forward through each step.. RNNs process tokens one at a time, left to right. Each step depends on the previous step's output, like reading a book one word at a time and never looking back.

This created three brutal bottlenecks. First, you can't parallelize. Step 47 has to wait for step 46, which has to wait for step 45, all the way back to step 1. Training was painfully slow. Second, the vanishing gradient problem meant that information from early tokens degraded through many sequential steps. An RNN reading a 500-word paragraph is essentially playing a game of telephone -- the message gets garbled over distance. Third, LSTMs (Long Short-Term Memory networks, introduced by Hochreiter and Schmidhuber in 1997) partially solved this with gating mechanisms, but still struggled with very long sequences.

Noam Shazeer, one of the Transformer co-authors, put it bluntly at NVIDIA GTC 2024:

"We could have done the industrial revolution on the steam engine, but it would just have been a pain. Things went way, way better with internal combustion."
-- Noam Shazeer

The transformer was that internal combustion engine. Instead of processing tokens sequentially, it processes the entire sequence at once. Self-attention lets any token attend to any other token directly -- no telephone game. Residual connections prevent vanishing gradients. The result: dramatically faster training and better handling of long-range dependencies.

"Attention Is All You Need"

The paper was published on June 12, 2017, at the 31st Conference on Neural Information Processing Systems (NeurIPS). The title is a reference to the Beatles song "All You Need Is Love." Jakob Uszkoreit, one of the authors, picked the name "Transformer" simply because he liked how the word sounded. An early internal design document was actually titled "Transformers: Iterative Self-Attention and Processing for Various Tasks" and included illustrations from the Transformers franchise.

The paper was notably understated. The authors weren't claiming to have solved AI; they were trying to build a better machine translation system. The base model had just 65 million parameters -- roughly 27,000 times smaller than GPT-4 -- and trained in 12 hours on 8 GPUs. It achieved 28.4 BLEU on English-to-German and 41.0 BLEU on English-to-French, both new state-of-the-art results. All eight authors were credited as equal contributors; the listed order was randomized.

Jensen Huang, speaking at GTC 2024 with the original authors, said simply: "Everything that we're enjoying today can be traced back to that moment."

Where the 8 Authors Went

Every single author left Google. They collectively went on to shape the AI industry:

As Llion Jones recalled: "We had very recently started throwing bits of the model away, just to see how much worse it would get. And to our surprise it started getting better." And Aidan Gomez offered a striking thought: "I think the world needs something better than the transformer."

Self-Attention: How the Model Decides What to Focus On

Self-attentionA mechanism allowing each token to attend to all other tokens in parallel, computing relevance scores between every pair. lets each word in a sentence look at every other word and decide how much to "pay attention" to each one, based on context. The word "bank" in "river bank" should attend heavily to "river"; in "bank account" it should attend to "account." This is how transformers resolve ambiguity.

The mechanism works through three learned vectors per token:

The math is elegant: for each token, compute how similar its Query is to every other token's Key (via dot product), scale by the square root of the dimension to keep gradients well-behaved, apply softmax to get a probability distribution, then take a weighted sum of all Values. The formula: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V.

Multi-Head Attention

Instead of running one attention computation, transformers run multiple attention heads in parallel -- each with its own learned Q, K, V projections. Why? Language has multiple simultaneous relationships. One head might learn to track syntax (subject-verb agreement), another semantics (meaning), another positional patterns (nearby words), another coreference (which noun a pronoun refers to). The original Transformer used 8 heads. GPT-3 uses 96.

Tokens and Tokenization

Language models don't see words. They see tokensThe atomic unit a language model processes -- a word, subword, or character mapped to an integer ID. -- subword fragments, each mapped to an integer. The dominant method is Byte Pair Encoding (BPE): start with individual characters, count all adjacent pairs in the training data, merge the most frequent pair into a single token, and repeat until you reach a target vocabulary size.

This creates some surprising consequences. The word "strawberry" gets tokenized as something like:

The model never sees individual letters. It literally cannot count the r's because "r" doesn't exist as a discrete unit in its perception. Numbers get split unpredictably too -- "123456" might become ["123", "456"] or ["12", "345", "6"]. Non-English languages often require more tokens per word, consuming more of the context window and costing more per interaction.

SentencePiece, developed by Google, takes a different approach: it treats input as a raw byte stream without assuming whitespace separates words. That's critical for languages like Chinese, Japanese, and Thai. It's used by T5, LLaMA, and most multilingual models.

Context Windows

The context windowThe maximum number of tokens a model can process at once -- its "working memory." is the model's working memory. Everything the model reads and writes must fit inside it. Tokens outside the window simply don't exist to the model.

The growth has been staggering -- roughly 20,000x in 8 years:

There's a catch: attention is O(n²) with sequence length. Double the context, quadruple the compute. And research shows models perform best on information at the beginning and end of the context -- the "lost in the middle" problem. The model's attention literally sags in the middle of long inputs, like a human reader's concentration on page 300 of a dense novel.

The Scaling Hypothesis

In March 2019, reinforcement learning pioneer Richard Sutton published a short, devastating essay called "The Bitter Lesson":

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
-- Richard Sutton, The Bitter Lesson (2019)

He called it "bitter" because researchers keep trying to build clever, hand-crafted solutions -- and they keep getting beaten by simpler methods that just use more compute. Chess (Deep Blue), speech recognition, Go (AlphaGo) -- the pattern repeats relentlessly.

This became the guiding principle of the AI boom. Ilya Sutskever, OpenAI co-founder, was its most prominent advocate: just make the model bigger, train it on more data, use more compute. In January 2020, Kaplan et al. at OpenAI quantified this. They found that model performance scales as a smooth power law with model size, data, and compute -- predictable across seven orders of magnitude. This led directly to GPT-3 at 175 billion parameters.

Then in March 2022, DeepMind's Chinchilla paper corrected the balance. Kaplan had overweighted model size; Chinchilla showed that model size and training data should scale equally. The practical rule: roughly 20 tokens per parameter. Their 70-billion parameter Chinchilla model, trained on 1.4 trillion tokens, outperformed 280-billion parameter Gopher trained on just 300 billion tokens -- despite being 4x smaller.

By 2025, the picture shifted again. Sutskever himself acknowledged that the original pre-training scaling recipe is reaching its limits because internet data is finite: "The age of scaling is giving way to the age of research."