The Transformer Architecture

§ 01 · HookThe Paper That Changed Everything

In 2017, a team of researchers at Google published a paper titled "Attention Is All You Need." It's one of the most influential papers in the history of AI. Every large language model you use today, whether it's Claude, GPT, Gemini, or Llama, is built on the architecture it introduced. Before that paper, the field was heading in a different direction. After it, almost everything changed.

This article isn't about memorising architecture diagrams; you can find those anywhere. It's about building real intuition: why each component exists, what problem it was designed to solve, and how the pieces fit together. By the end, you should be able to explain the Transformer to a friend in plain English without handwaving.

We'll use some mathematical intuition (dot products, softmax, weighted sums) but no equations. Code comes in later articles. Today is about the mental model, and as always, we start with the problem that needed solving.

§ 02 · The ProblemWhy the Transformer Was Invented

To appreciate a solution, you have to feel the pain of the problem first. Before Transformers, the standard way to process text was the Recurrent Neural Network, or RNN, and its more capable cousin, the LSTM (Long Short-Term Memory).

How RNNs worked

Imagine reading a sentence where you're only allowed to look at one word at a time, and after each word you have to summarise everything you've understood so far on a single sticky note before moving on. That sticky note is the hidden state.

An RNN processes tokens sequentially, left to right. It reads "The" and updates its hidden state. Then it reads "cat" and updates the hidden state again. Then "sat", "on", "the", "mat". By the time it reaches "mat", the hidden state is supposed to hold the model's compressed understanding of the whole sentence.

The trouble is that language isn't a conveyor belt. Words relate to other words that are far away, not only to the word right before them. That caused two big problems.

Problem 1: sequential processing kills parallelism

Each step depends on the output of the previous step, so you can't process tokens in parallel. You have to finish step 1 before you can start step 2. A modern GPU has thousands of cores that could be working at once (Article #01, §05), and an RNN leaves almost all of them idle. Training on long sequences was painfully slow.

Problem 2: vanishing gradients destroy long-range memory

Remember backpropagation from Article #01, the restaurant manager tracing a complaint backwards through every station? In an RNN, that backward trace has to pass through every single timestep. Over a long sequence, the gradient gets multiplied by the same weights again and again, and if those numbers are smaller than 1, the signal shrinks exponentially. By the time you're trying to connect a word to something 50 words earlier, the gradient is essentially zero, so the network can't learn the relationship.

This is the vanishing gradient problem. LSTMs were invented to address it. They added memory gates, small learned switches that decide at each step what to keep, what to forget, and what new information to add to the hidden state. Even so, LSTMs struggled with very long documents, and the sequential bottleneck stayed.

Think of the game of telephone, where a message is whispered from person to person down a long line. Each handoff loses a little detail, and by the 50th person the original sentence is unrecognisable. In an RNN, the "message" is the training signal travelling backwards from word 50 to word 1, and it degrades the same way.

The Transformer's radical proposal

The paper's authors made a bold bet: throw away recurrence entirely. Instead of processing tokens one at a time, process them all at once, and use a mechanism called attention that lets every token relate directly to every other token in a single step, however far apart they are.

The result is that training can be spread across many GPUs at once, and any two tokens can influence each other directly. The information from "trophy" no longer has to survive a seven-step relay to reach "it". It arrives in one step, weighted by how relevant the attention mechanism decides it is.

§ 03 · InputsTokens, Embeddings, and Positions

Before we get to attention, let's look at what actually goes into the Transformer. The model doesn't see text, and it doesn't see letters. It sees numbers, specifically dense vectors: long lists of decimal numbers where most positions carry some meaning. (The opposite is a "sparse" representation such as one-hot encoding, where almost every position is zero.)

Step 1: tokenisation

First, the text is broken into tokens. In Article #01 we said you could think "token ≈ word". Here's the more precise picture. Most modern LLMs use subword tokenisation, usually a method called Byte-Pair Encoding (BPE). BPE notices which chunks of letters appear constantly in its training text, like "ing" or "tion", and gives those chunks their own tokens, while rare strings get chopped into smaller pieces. Common words may be a single token; rare or long words get split.

For example, "Hello" is usually one token, while a longer word like "unbelievable" might be split into two or three pieces. Code, numbers, and unusual words tend to break into more pieces. Article #03 goes deep on this. For now, just know that text becomes a sequence of integer IDs, one per token.

Step 2: token embeddings

Each token ID is looked up in an embedding table, a big matrix with one row per token. Each row is that token's vector. If a model's vocabulary (the fixed set of tokens it knows) has 50,000 tokens and each embedding has 768 numbers, the table is a 50,000 × 768 matrix. (GPT-2 is almost exactly this size: 50,257 tokens and 768 numbers each in its smallest version.)

These numbers encode meaning. Tokens with similar meanings end up with similar vectors: the vector for "king" sits closer to "queen" than to "car", and closer still to "monarch", because those words appear in similar contexts. The table isn't written by hand. It's learned during training, like every other parameter. Article #04 is entirely about embeddings.

Step 3: positional encoding

Here's a subtle problem. The Transformer looks at all the tokens at the same time, so unlike an RNN it has no built-in sense of order. "Dog bites man" and "man bites dog" would arrive as the same set of vectors, and the model wouldn't know which word came first.

The fix is positional encoding. Before the tokens go in, each token's embedding gets a small extra vector added to it that encodes its position in the sequence. It's like stamping each exam paper with the student's seat number before handing the whole pile over at once.

The original paper built these position vectors from fixed sine and cosine waves at different frequencies. Many later models learn their position vectors during training instead, and most recent LLMs use a scheme called Rotary Position Embeddings (RoPE), which rotates parts of each vector by an angle that depends on position rather than adding a separate vector. Article #05 covers RoPE properly. The principle is the same in every case: inject position information so the model knows the order of tokens even though it sees them all at once.

§ 04 · The Core MechanismThe Attention Mechanism

This is the most important section of the article, so take your time with it. We'll go from intuition to mechanism one step at a time.

The intuition: your brain already does this

Read this sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? The animal. You knew instantly, without effort. Your brain scanned back through the sentence, weighed what was relevant, and settled on "animal". You attended to the right word.

Attention in a Transformer is a mathematical version of that process, and it's the "highlighter" from Article #01 made precise. For each token, the model asks: "how relevant is every other token to understanding this one, right now?" Tokens with high relevance get high attention weights and contribute more to the result. Tokens with low relevance are mostly ignored.

Query, Key, and Value

This is where many explanations lose people. The Q-K-V idea sounds abstract, but a search engine makes it concrete.

In practice, each token's vector is turned into three new vectors by multiplying it by three separate learned matrices. This is the same weighted-sum arithmetic that happens inside every neuron (Article #01, §05), just done three times with three different sets of weights:

• Query (Q): "What am I looking for?"
• Key (K): "What do I contain?"
• Value (V): "What do I contribute if you pick me?"

The attention computation, step by step

For one token, say "it", the model does four things:

1. Score. Take the dot product of the Query vector for "it" with the Key vector of every token in the sequence. A dot product multiplies matching positions in two vectors and adds up the results, giving one number that says how well the two vectors line up. Vectors pointing in a similar direction give a large number; unrelated ones give something near zero. High score means high compatibility.
2. Scale. Divide every score by the square root of the key vector's length (written √d_k). Without this, scores in big models get so large that the next step pushes nearly all the weight onto one token, and the gradients shrink towards zero, which is the vanishing-gradient problem from §02 turning up in a new place.
3. Softmax. Turn the scaled scores into weights that add up to 1, using the same softmax you met in Article #01, §08. Think of it as converting raw scores into percentages that sum to 100%.
4. Blend. Add up all the Value vectors, each multiplied by its weight. Picture a smoothie made from every token's Value vector, where highly attended tokens get poured in generously and the rest contribute barely a drop.

The result is a new, richer vector for "it", one that has absorbed information from every relevant token. The word "it" now "knows" about "animal", because attention blended the Value vector of "animal" into its representation with a large weight.

Multi-head attention

One attention operation tends to capture one kind of relationship. But language is rich, and tokens relate to each other in several ways at once.

Multi-head attention runs several independent attention operations side by side. Each "head" has its own Q, K, and V matrices, so it can learn to look for different patterns. The outputs of all the heads are joined together (concatenated) and passed through one more learned matrix to produce the final result.

The smallest GPT-2 has 12 attention heads in each layer, and GPT-3 has 96. Nobody tells a head what to specialise in. Each one works out a useful pattern during training, the same way the feature detectors in Article #01, §05 emerged without being programmed. (Article #05 goes much deeper into attention, including what researchers find when they look inside real heads.)

§ 05 · The ArchitectureThe Full Transformer Block

Attention is the heart of the Transformer, but it isn't the whole thing. A Transformer is a stack of identical blocks (also called layers), and each pass through a block refines the token vectors a little more. Each block has four parts, always in the same order.

Part 1: multi-head self-attention

Each token looks at the others, weighted by learned relevance, exactly as in §04. The output is a new vector for each token, informed by the whole context.

Parts 2 and 4: Add & Norm

This step has two pieces. The first is the residual connection: the input to the attention step is added back to its output (x + Attention(x)). The idea comes from ResNets (Residual Networks), an image-recognition architecture from 2015 that made this shortcut famous.

Think of it as a bypass lane on a highway. The training signal can skip straight past a layer and flow backwards, so even if one layer is struggling to learn anything useful, the earlier layers still get a clear signal. That's what makes very deep networks trainable. It also means each block only has to learn "what to change" about its input, rather than rebuilding the whole output from scratch, which is a much easier job.

The second piece is layer normalisation. It rescales the numbers flowing through each layer so they stay in a steady, well-behaved range (centred around zero, without wild swings in size). Without it, values can drift towards extremes as they pass through dozens of layers, and training becomes unstable.

Part 3: the feed-forward network (FFN)

Once attention has decided which tokens relate to which, a small two-layer neural network, the same kind of layered network you met in Article #01, §05, is applied to each token separately. It expands the vector to a larger size, applies a non-linear "bend", then shrinks it back. GPT-2 uses GELU for the bend, a smoother cousin of the ReLU from Article #01, and many newer models use related variants.

Research suggests the FFN layers are where much of a model's stored knowledge lives: facts, domain patterns, common associations. Attention decides which tokens relate to each other; the FFN works out what to do with that information. The two complement each other.

After N passes through this block, the token vectors are rich and deeply shaped by context. That's 12 blocks in the smallest GPT-2 and 96 in GPT-3.

§ 06 · Model FamiliesEncoder, Decoder, or Both

The original Transformer was designed for machine translation, which takes one sequence in and produces another sequence out. It had two stacks of blocks: an encoder to understand the input sentence, and a decoder to generate the translation. Since then, people have found that many jobs only need one of the two stacks, which gives three families of models.

Encoder-only (for example BERT)

These models read and understand text. Attention is bidirectional: every token can look at every other token, before and after it. They're excellent for classification, for named entity recognition (automatically finding and labelling the names of people, places, and organisations in text), and for producing embeddings. They aren't built to generate text.

Decoder-only (for example GPT, Llama, Mistral)

These models generate text one token at a time, the autoregressive loop from Article #01, §08. Attention is causal, also called masked: each token can only look at the tokens before it. That's essential, because the model is trained to predict the next token, and if it could see the future there would be nothing to learn. It's the same reason your phone's predictive text can't peek at the word you haven't typed yet.

Encoder-decoder (for example T5, BART)

These models read the whole input with a bidirectional encoder, then generate the output with a causal decoder that also looks back at the encoder's results. They suit tasks where the input and output are clearly separate texts, such as translation and summarisation.

§ 07 · ScaleScale and Why It Matters

One of the most striking facts about modern AI is that the core architecture from the 2017 paper is still recognisably the one inside today's frontier models. Details have been refined (where the norms sit, how positions are encoded, which activation is used), but what really changed was scale.

The Transformer's parallel design is what makes this scaling possible. Because every token in a sequence is processed at once, the work can be spread across thousands of GPUs. More chips, more data, bigger models: the architecture keeps scaling where RNNs stalled.

Emergent abilities

As Article #01, §16 described, bigger models trained on more data have shown abilities nobody explicitly trained for: multi-step arithmetic, chain-of-thought reasoning (writing out steps before answering), analogies, and in-context learning, where a model picks up a new task from a few examples in the prompt without any change to its weights.

Researchers still debate how sudden these jumps really are, since part of the apparent suddenness can come from all-or-nothing scoring. Either way, nobody designed these abilities in. They came as a by-product of learning to predict text at enough scale, and the Transformer turned out to be an unusually good architecture for soaking up that structure.

Same architecture, very different model

What turns a raw, pre-trained Transformer into the helpful assistant you chat with? You met the answer in Article #01, §12: supervised fine-tuning on curated instruction-response pairs, then preference training such as RLHF. None of it changes the architecture. It changes what the model has learned to do with it. Phase 2 of this series (Articles #11 and #12) shows how fine-tuning is actually done. Next, though, we zoom in on the very first step of the pipeline: tokenisation.

§ 08 · SummaryThe Mental Model You Now Have