Attention, Step by Step: The Transformer Mechanism by Hand

Most explanations of attention are a diagram and a formula. This one is arithmetic. Two words go in, the cat, and every intermediate number is printed. A script computed them and asserts its own results; nothing here is illustrated.

What attention does, in one line

Every word rewrites itself as a weighted blend of the words it can see.

The whole game is choosing those weights. Nobody writes a rule for them; they fall out of arithmetic on the vectors themselves, and what drops out is a handful of numbers that add up to 1.

The arrows only ever go backwards, or curl round to the word itself. That is causal masking: a word may look at what came before it and never at what comes after.

Query, key, value

Each word turns its one vector into three, and the database names are worth taking literally:

Scoring word i against word j is the dot product of i's query with j's key: no lookup table, no comparison logic, just how well two arrows line up. Those scores become percentages, and every word pays out its value in those proportions.

Hold on to that sentence. Every attention variant you will ever read about (multi-head, grouped-query, FlashAttention) keeps it exactly as it is. They argue about bookkeeping, not about what attention means.

The calculation

Two words in, one word out. Everything below is that.

We start from Q, K and V. Each is the token vector times a learned matrix, and that multiplication is not where the interesting part lives. Assume it has happened:

Q  what each word asks        q0       q1       q2       q3
            the                        0.300   -0.200    0.500    0.100
            cat                        1.100   -0.800    0.200    0.300

            K  what each word offers      k0       k1       k2       k3
            the                       -0.400   -0.300    0.200    1.200
            cat                        1.600   -0.700   -0.400   -0.500

            V  what each word gives       v0       v1       v2       v3
            the                       -0.500   -0.400    0.100    1.100
            cat                        1.200   -0.300   -0.200   -0.600

Only cat matters from here. At generation time only the last word predicts; during training every row predicts its own next token, but for this walkthrough everything we do to the is bookkeeping.

Walk it yourself, one step at a time:

Step 1: score

Dot each query with each key. cat against itself:

q_cat · k_cat = (1.100 ×  1.600) + (-0.800 × -0.700)
                          + (0.200 × -0.400) + ( 0.300 × -0.500)

                          =  1.76 + 0.56 - 0.08 - 0.15  =  2.09

And cat against the:

q_cat · k_the = (1.100 × -0.400) + (-0.800 × -0.300)
                          + (0.200 ×  0.200) + ( 0.300 ×  1.200)

                          = -0.44 + 0.24 + 0.04 + 0.36  =  0.20

2.09 against 0.20. Ten times the match, and nothing produced that except two rows of numbers agreeing or not.

scores          the      cat
            the           0.160    0.370
            cat           0.200    2.090

Step 2: scale

Divide by √d_k = 2.

scaled          the      cat
            the           0.080    0.185
            cat           0.100    1.045

A dot product adds up head_dim separate products, and their spread grows with the square root of head_dim. At head_dim 128 the raw numbers would be about 5–6× larger, the softmax below would collapse onto one option, and the gradient trying to flow back through it would be roughly zero. That square root is exactly why the divisor is √d_k and not d_k.

Step 3: mask

the comes first. It cannot see cat, which comes after it.

masked          the      cat
            the           0.080     -inf
            cat           0.100    1.045

−∞ rather than a small number, because the next step exponentiates and e^-∞ is exactly 0. A masked position gets no weight at all, not a little.

Step 4: softmax

Exponentiate the cat row, then divide by its total.

e^0.100 = 1.1052        1.1052 / 3.9486 = 0.2799
            e^1.045 = 2.8434        2.8434 / 3.9486 = 0.7201
                      ──────                          ──────
              total =  3.9486                          1.0000

cat gives 72% of its attention to itself and 28% to the. That is the sentence's shape falling out of arithmetic: a determiner carries little meaning, so it gets the smaller share, and nobody wrote a rule saying so.

Notice that 28% is not nothing. Attention is a smear, not a pointer. Apart from the cells the mask kills outright, every word puts some weight on every word it can see, and there is no such thing as "the model attends to word 2". There is a distribution, and it is usually softer than you would expect.

Step 5: mix

Spend those weights on the value vectors.

0.2799 × V[the] = -0.1399  -0.1120   0.0280   0.3079
            0.7201 × V[cat] =  0.8641  -0.2160  -0.1440  -0.4321
                              ────────────────────────────────────
            out[cat]        =  0.7242  -0.3280  -0.1160  -0.1242

cat walked in as [1.200, -0.300, -0.200, -0.600] and walked out as [0.724, -0.328, -0.116, -0.124], carrying a 28% share of the that it did not have before.

That is the whole job of attention. One word's vector now contains part of another's.

And then?

The blended vector is what the rest of the network sees. In the walkthrough above, step 6 scores it against a three-word vocabulary and sat takes 55%. Treat that step as a demonstration, not a result: the vocabulary matrix there was chosen to make the toy readable, a real model reaches its vocabulary through a residual, a LayerNorm and an MLP first, and next-token prediction is a different mechanism from the one this article is about. Attention's job ended when out[cat] came out of step 5.

We chose these numbers so the example would come out legible. Real Q, K and V come from weight matrices that start random and get beaten into shape by training, and their rows look like nothing at all until they do. The arithmetic here is exactly the arithmetic a real model runs; only the tidiness is ours.

The mask, on a longer sentence

Two words make a small picture. Here is the same rule on four, the cat sat on. It is a fresh set of Q, K and V, so the weights are new; only the arithmetic carries over:

The hatched cells are the mask: each word may only spend weight on itself and what came before it. The two worked rows run the same softmax you just ran, exponentiating each live score and dividing by the row total, but on this run's own numbers. None of these digits are the ones you computed above. Row 2 splits 0.784/0.216 where ours split 0.28/0.72; different weights, same rule. What carries over is the shape: nothing above the diagonal gets any weight, and every row still adds up to 1.

More than one head

Everything computed on this page is one head of head_dim 4. That is what the Q, K and V tables above are. Real models run a crowd of heads in parallel, each with its own W_Q, W_K and W_V, each reading the whole token vector and writing a head_dim slice of the output. One head can only ask one kind of question; ours asked what came before me? A second head, reading the same words, can ask what frames this? and land somewhere else entirely.

Splitting the model dimension into n_heads slices costs nothing extra in FLOPs and buys several independent questions per layer. A four-head layer of the shape above would have d_model 16 and four such blocks side by side, each doing exactly the arithmetic you just did, on its own slice. That is where the next article starts, because four heads is also where the cache starts to hurt.

Check it yourself

One script sits behind this article and it self-asserts: predict_sat.py recomputes every table above, including the prediction, and refuses to run if a number drifts. (The four-token grid comes from its companion, attention_numbers.py.)

There is nothing in it you cannot redo on paper. Take the cat row of Q, dot it with each row of K, halve, exponentiate, divide by the total: 0.2799 and 0.7201. That is attention, and the rest of a transformer is built on top of it.

What this leaves out

Nothing here is about cost. Real serving stacks cache K and V for every token they have generated, and that cache, not the weights, is usually what decides how many users a GPU can hold. Grouped-query attention, FlashAttention and multi-head latent attention all attack that bill, and none of them changes the five steps above. That is the next article in this series: grouped-query, multi-query, FlashAttention and multi-head latent attention, all run on four tokens with the bill printed for each. If you want the version of this problem that is already in production, we took apart TurboQuant, which compresses the same cache without touching the architecture.

Frequently Asked Questions

Why divide by the square root of d_k?

A dot product sums head_dim separate products, and the spread of that sum grows with the square root of head_dim, not with head_dim itself. Dividing by √d_k keeps the scores in a range where softmax stays soft. Without it, at head_dim 128 the raw scores would be roughly 5–6× larger, softmax would collapse onto a single option, and the gradient flowing back through it would be about zero.

Why mask with negative infinity instead of zero?

Because the next step exponentiates. e^0 is 1, which is a substantial weight; e^-∞ is exactly 0. Masking with zero would leave every forbidden position with a real share of the attention. Masking with −∞ removes it completely.

Does attention really only look backwards?

In a decoder-only language model, yes. That is what causal masking enforces, and it is why the model cannot cheat during training by reading the answer. Encoder models such as BERT use the same arithmetic with no mask, so every word sees every other word in both directions.

What is the difference between head_dim and d_model?

d_model is the width of the vector each token carries between layers. head_dim is the slice of it one attention head works on: a layer with n_heads heads splits d_model into n_heads slices of head_dim each. This article runs one head of head_dim 4, so d_model and head_dim happen to coincide.

Is this the same arithmetic a real model runs?

Yes. The dot product, the √d_k division, the mask, the softmax and the weighted sum are exactly what a production attention kernel computes. What is artificial here is the numbers: real Q, K and V come out of trained weight matrices and look like nothing in particular. Only the tidiness is ours.

Further reading

• Vaswani et al., Attention Is All You Need (2017). The original, and still the clearest statement of the scaling term

About J33.AI

At J33.AI we build AI systems people have to trust, which means understanding them down to the arithmetic. Our AI foundations work starts where this article does: with what the model is actually computing, not with what the diagram suggests. More about how we work.