Transformers are often explained as a list of parts: embeddings, attention and feed-forward layers. A list like that does not show how the parts work together. This article takes a different approach. We follow one four-token sentence through a complete block, track the tensor shape at every step, and compute one row of attention by hand. We also separate the three model families and show how training differs from generation. By the end, you should be able to trace a token from an integer ID to a probability.
Key takeaways
- A transformer is a stack of blocks with the same structure. Each block runs attention, where positions exchange information, then a feed-forward network, where each position is transformed on its own. Residual connections and normalization surround both.
- Attention is a weighted sum of value vectors. The weights come from softmax(QKᵀ/√dₖ). It is a learned calculation, not a human-like spotlight.
- Attention does not know word order by itself, so position information must be added.
- There are three layouts. The original design is encoder-decoder. Decoder-only models use masked self-attention and no cross-attention. Encoder-only stacks build representations of an input.
- Training processes all positions in parallel under a causal mask. Generation still happens one token at a time, and self-attention cost grows with the square of sequence length.
What is the transformer architecture?
A transformer is a neural network that processes sequences of tokens mainly through attention and position-wise feed-forward layers, rather than recurrence or convolution. Self-attention lets each token build a weighted mix of information from other tokens in the context. The design was introduced in the 2017 paper "Attention Is All You Need" (arXiv). It is best understood as a family of building blocks, not one fixed model.
The original paper describes a model based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. It used separate stacks of six encoder layers and six decoder layers (Attention Is All You Need, arXiv). Later models vary that recipe. They can differ in how they encode position, where they place normalization, and which tokens may attend to which.
Vocabulary you need first
Beginners often blur the different kinds of data inside a transformer. Keep these six apart:
| Term | What it is | Example shape |
|---|---|---|
| Token | An integer ID for a piece of text | (B, T) |
| Embedding | A learned vector looked up for that ID | (B, T, d_model) |
| Hidden state / residual stream | The per-position vector each block reads from and adds to | (B, T, d_model) |
| Logits | Raw, unnormalized scores, one per vocabulary entry | (B, T, V) |
| Probabilities | Softmax of the logits; they sum to 1 | (B, T, V) |
| Generated token | The ID picked or sampled from those probabilities | (B, 1) |
Here B is batch size, T is the number of tokens, d_model is the vector width and V is the vocabulary size.
Is attention really all you need?
No. The title emphasizes attention, but it does not describe the whole model. Every block in the original design also has a feed-forward sublayer, residual connections and layer normalization. The model also needs embeddings and position information. The feed-forward sublayer has a specific job. It applies the same nonlinear transformation to each position independently, after attention has mixed information across positions (arXiv). Attention alone does not supply that position-wise processing.
Be careful with the phrase "the model understands meaning." A transformer learns vector representations that help it reach a training goal, such as predicting the next token. Whether those representations amount to understanding is a separate question.
Why are transformers preferred over RNNs for many language tasks?
A central reason is training speed on parallel hardware. A recurrent neural network (RNN) reads tokens one at a time, so each step must wait for the previous one. A transformer can process every position of a training sequence together. It also gives any two tokens a direct attention path, so distant words are one step apart instead of many.
An RNN keeps a single hidden state, a running summary vector. It updates that state token by token. Position 50 depends on position 49, which depends on 48, and so on. That chain cannot be split across positions during training. It also forces information from early tokens to survive many updates.
A transformer removes that chain during training. The original authors reported that their model was more parallelizable and needed significantly less training time than the sequence-transduction models they compared against (Google Research). That result describes the models in their comparison, not every possible RNN.
The trade-off is real, though:
| Property | RNN | Transformer |
|---|---|---|
| Path between two distant tokens | Many sequential steps | One attention step |
| Training across positions | Sequential | Parallel |
| Text generation | One token at a time | Still one token at a time |
| Memory as input grows | Fixed-size state | Grows with length; attention scores grow with T² |
Think of an RNN as a relay race: the baton passes from runner to runner. A transformer is a meeting where everyone can speak to everyone directly. The meeting avoids the relay's delays, but the number of possible conversations grows quickly with the number of people.
How does a transformer work, step by step?
A transformer converts token IDs to vectors, adds position information, then sends them through a stack of blocks. Each block lets tokens share information through attention, then transforms each token with a feed-forward network. Residual connections add each result back to a running vector. After the last block, a linear layer produces a score for every vocabulary entry, and softmax turns scores into probabilities.
We will use a tiny, illustrative configuration. The sentence "the cat sat down" becomes four tokens. Batch size B = 1, T = 4, d_model = 8, h = 2 heads, so each head has d_k = 8 / 2 = 4. Assume a vocabulary of V = 50,000. Real models are far larger, but the shapes follow the same pattern.
- Tokenize. Text becomes integer IDs. Shape: (1, 4).
- Embed. Each ID selects a row from a learned table. Shape: (1, 4, 8).
- Add position. Attention alone does not encode order, so the original Transformer added positional encodings to the embeddings (arXiv). Shape stays (1, 4, 8). This sum starts the residual stream.
- Project to queries, keys and values. Three learned matrices turn each vector into a query (what this position is looking for), a key (what this position offers) and a value (the content it passes on). Each is (1, 4, 8), then split into heads: (1, 2, 4, 4), meaning (batch, heads, T, d_k).
- Score. Each query is compared with every key using a dot product, then divided by √d_k = 2. Shape: (1, 2, 4, 4), meaning one 4×4 score grid per head.
- Mask. In a decoder, scores for future positions are set to negative infinity. Each token can then see only itself and earlier tokens.
- Weight and combine. Softmax turns each row of scores into weights that sum to 1. The weights multiply the values: softmax(QKᵀ/√dₖ)V. Shape: (1, 2, 4, 4), meaning (batch, heads, T, d_k). It matches step 5 only because T and d_k are both 4 here.
- Merge heads. The two heads are concatenated back to width 8 and passed through one more linear layer. Shape: (1, 4, 8). Multi-head attention runs several attention functions in parallel on different learned projections and then combines them (arXiv).
- Add and normalize. The attention output is added to the residual stream, then layer normalization rescales each vector. Shape: (1, 4, 8).
- Feed-forward. Each position passes through the same two-layer network independently. In the original design, it widens to a larger hidden size, applies a ReLU nonlinearity, then narrows back. Shape: (1, 4, 8). Add and normalize again.
- Repeat. Steps 4 to 10 form one block. The model stacks several blocks; the original used six per stack.
- Produce logits and probabilities. A final linear layer maps each vector to vocabulary scores. Shape: (1, 4, 50,000). Softmax gives probabilities. The last row predicts the token after "down".
What do residual connections and layer normalization actually do?
A residual connection adds a sublayer's output to its input instead of replacing it. Picture a shared notebook passed through the stack. Each block reads the notebook and writes notes in the margin, but never erases the page. This gives information and gradients a direct route through a deep stack. Gradients are the error signals used to adjust the weights. Layer normalization rescales each vector to a consistent range, which helps keep training stable.
The original paper computed each sublayer's output as LayerNorm(x + Sublayer(x)), placing normalization after the residual addition (arXiv). Some later implementations normalize before each sublayer instead. This is one of the implementation details that varies between models.
The block as pseudocode
# x: residual stream, shape (B, T, d_model)
def decoder_block(x, mask):
x = layer_norm(x + multi_head_attention(x, x, x, mask)) # tokens exchange info
x = layer_norm(x + feed_forward(x)) # each token transformed
return x
x = embed(token_ids) + positions(token_ids) # (B, T, d_model)
for block in blocks:
x = decoder_block(x, causal_mask)
logits = x @ W_vocab # (B, T, V)
probs = softmax(logits, axis=-1)This sketch is a decoder-only block. It follows the original post-norm ordering and omits dropout for clarity. A decoder in the original encoder-decoder design would also have a cross-attention sublayer.
How does attention work? A worked example with real numbers
Attention computes, for each token, a weighted average of value vectors. The weights come from how well that token's query matches every key, after scaling and a softmax. A causal mask forces future positions to zero weight. The worked example below computes one full row by hand. It shows that attention is ordinary arithmetic, not a mysterious spotlight.
Illustrative numbers only. Take the third token, "sat", in one head with d_k = 4. Suppose its query's dot products with the four keys are:
| Key position | "the" | "cat" | "sat" | "down" |
|---|---|---|---|---|
| Raw score q·k | 2 | 4 | 6 | (future) |
| Divided by √4 = 2 | 1 | 2 | 3 | −∞ (masked) |
| eˣ | 2.718 | 7.389 | 20.086 | 0 |
| Weight (÷ 30.193) | 0.090 | 0.245 | 0.665 | 0 |
The masked position contributes nothing because e^(−∞) = 0. Now use tiny two-number value vectors: "the" = [1, 0], "cat" = [0, 1], "sat" = [1, 1]. Real value vectors in this head would have d_k = 4 numbers each. Two numbers keep the arithmetic short.
Output = 0.090×[1, 0] + 0.245×[0, 1] + 0.665×[1, 1] = [0.755, 0.910].
That vector is the new information for "sat" from this head. The paper describes exactly this: a weighted sum of values, with weights set by query-key compatibility (arXiv).
Why divide by √d_k?
The original authors suspected that for large d_k, dot products grow large in magnitude. Large inputs push softmax toward one weight near 1 and the rest near 0, where gradients become extremely small. Dividing by √d_k counteracts this effect (arXiv).
A lookup analogy, and its limit
Queries, keys and values resemble a library search. The query is your search phrase. Keys are the catalogue labels. Values are the books. The difference is that attention never picks just one book. It blends all of them according to match strength. A plain-language overview of this idea appears in IBM's explainer on attention mechanisms.
Encoder decoder transformer, encoder-only or decoder-only: what is the difference?
The three transformer families differ in which attention types they use. Encoder-only models let every token see every other token, which suits building representations. Decoder-only models mask future tokens and generate text one token at a time. Encoder-decoder models combine both: an encoder reads the input, and a decoder generates output while consulting the encoder through cross-attention. The original 2017 Transformer was encoder-decoder.
The original paper used multi-head attention in three distinct ways (NeurIPS proceedings):
- Encoder self-attention. Every input position can attend to every position in the previous encoder layer.
- Masked decoder self-attention. Each position attends only to itself and earlier positions.
- Encoder-decoder (cross) attention. Queries come from the decoder. Keys and values come from the encoder output.
| Family | Attention used | Typical tasks | Commonly cited example |
|---|---|---|---|
| Encoder-only | Bidirectional self-attention | Building representations for classification or tagging | BERT |
| Decoder-only | Masked self-attention, no cross-attention | Open-ended text generation | GPT series |
| Encoder-decoder | Encoder self-attention, masked decoder self-attention, cross-attention | Translation and other sequence-to-sequence tasks | Original Transformer |
Two common errors follow from this table. First, not every transformer is encoder-decoder. Second, not every decoder uses cross-attention. A decoder-only model has no encoder output to attend to. Libraries also let you pair an autoencoding encoder with an autoregressive decoder for sequence-to-sequence tasks (Hugging Face documentation).
In translation, the encoder represents the source sentence. The decoder writes the translation one token at a time, using cross-attention to consult the source representation (NeurIPS proceedings).
How do training and generation differ?
During training, a decoder sees the whole correct sequence at once and predicts every next token in a single parallel pass. The causal mask stops each position from seeing the token it must predict. During generation, no correct sequence exists. The model must predict one token, append it, and run again. This is why a transformer trains in parallel but still generates sequentially.
Training: teacher forcing under a causal mask
Teacher forcing means feeding the true previous tokens as input, not the model's own guesses. If "the cat sat down" sits inside a longer training text, one forward pass yields four predictions:
- From "the", predict "cat".
- From "the cat", predict "sat".
- From "the cat sat", predict "down".
- From "the cat sat down", predict the next token in the training text.
The mask keeps each prediction from seeing its own answer. A common loss is cross-entropy on the correct next token, averaged across positions. Exact objectives vary by model family. Encoder-only models, for example, are not trained by next-token prediction in this left-to-right way.
Generation: the autoregressive loop
At inference, the model runs, reads the last row of probabilities, and picks a token. It may take the most likely one or sample. It appends that token and repeats. Each step depends on the previous one, so this part cannot be parallelized across new tokens. Implementations commonly store earlier keys and values in a cache so they are not recomputed at every step.
What does attention cost?
Self-attention compares every token with every other token, so a sequence of T tokens produces a T × T score grid per head per layer. Double the length and the grid grows fourfold. The original paper lists self-attention's per-layer complexity as quadratic in sequence length (arXiv). This growth is one major constraint on how long a context a model can process cheaply, and it motivates work on more efficient attention variants.
Illustrative numbers. At T = 1,000, each head in each layer computes 1,000,000 scores. At T = 2,000, it computes 4,000,000. Multiply by many heads and many layers, and the cost becomes substantial. The number of feed-forward operations, by contrast, grows linearly with T, because each position is processed independently.
A long context window also does not guarantee the model uses every token well. Behaviour depends on training, the position scheme, optimization and how information interferes inside the model.
When is a transformer the wrong choice?
A transformer is not the right tool for every sequence problem. Its training parallelism matters most when you have substantial data and parallel hardware. It is a weaker fit when sequences are extremely long, memory is tight, data is scarce, or a fixed-size running state is required. In those cases, simpler models, recurrent designs or efficient-attention variants are worth testing.
Consider alternatives when:
- Data is small. Transformers typically have many parameters, so training from scratch on little data is risky. A pretrained starting point or a smaller model may work better.
- The input is small tabular data. A sequence architecture may be unnecessary; test simpler established methods first.
- Sequences are very long and memory is fixed. Quadratic attention can exhaust memory.
- You need constant memory per step, as in some streaming or embedded systems, where an RNN's fixed-size state is an advantage.
- You need faithful explanations. Attention weights are one intermediate step. They do not by themselves give a complete causal explanation of a model's prediction.
What are the most common mistakes when learning transformers?
Many mistakes come from blurring distinctions: tokens versus embeddings versus logits, self-attention versus cross-attention, and training versus generation. Others come from assuming the original 2017 design describes every later model. Use this checklist to test your own mental model before you read papers or code.
- Do I know which tensor I am talking about: token, embedding, hidden state, logit or probability?
- Do I remember that position information must be added, because attention alone does not impose order?
- Can I say whether a layer uses self-attention, masked self-attention or cross-attention?
- Do I know that decoder masking hides future target tokens, while cross-attention can still read the full encoder output?
- Am I treating attention as weighted arithmetic rather than a spotlight?
- Am I separating parallel training from sequential generation?
- Am I assuming that six layers, post-norm and the original positional encodings apply to every model? They do not.
What to learn next
No related posts on this site are available yet, so the best next step is the primary source. Read the original paper, Attention Is All You Need on arXiv. Pay particular attention to its architecture figure and the section on why self-attention was chosen. Then compare that design with how libraries combine encoders and decoders in practice, using the Hugging Face encoder-decoder documentation. Try writing the tensor shape beside each step as you read. If you can do that, you can trace the architecture from input to output instead of treating it as a list of parts.

