LEARN · LLMS & TRANSFORMERS
Why self-attention matters
Transformers power modern large language models, image generators, coding assistants, and multimodal AI systems. The core idea behind them is surprisingly compact: self-attention lets every token decide which other tokens are most relevant before producing its next representation.
Unlike older sequence models that process text one step at a time, transformers process an entire sequence in parallel. Self-attention is the mechanism that makes this possible while still allowing every word to “look at” every other word.
By the end of this guide, you’ll understand:
-
Why Queries (Q), Keys (K), and Values (V) exist
-
How attention scores are computed
-
Why scaling and softmax are necessary
-
What multiple attention heads actually do
-
A tiny runnable NumPy implementation
-
How this connects to real LLMs
The intuition
Imagine reading this sentence:
“The animal didn’t cross the road because it was too tired.”
When you read the word “it”, your brain naturally connects it with “animal”, not “road”.
Self-attention performs a similar operation.
Every token asks:
“Which other tokens should I pay attention to before updating my meaning?”
Instead of using hand-written rules, transformers learn this behavior during training.
The three ingredients: Q, K, and V
Every token is transformed into three vectors:
-
Query (Q) — what this token is looking for
-
Key (K) — what this token offers
-
Value (V) — the information this token contributes
Think of it like a search engine.
| Component | Analogy |
|---|---|
| Query | Search request |
| Key | Search index |
| Value | Actual document |
Each token compares its Query against every Key.
If a Key matches well, its Value becomes important.
Step 1: Start with embeddings
Suppose we have three tokens:
Cats chase mice
Each token begins as an embedding.
Cats → embedding chase → embedding mice → embedding
These embeddings are simply vectors.
For example:
Cats = [0.2, 0.8, 0.4] chase = [0.5, 0.1, 0.7] mice = [0.9, 0.3, 0.2]
Real models typically use dimensions such as 768, 1024, 4096, or more.
Step 2: Create Q, K, and V
The transformer learns three matrices:
-
WQ
-
WK
-
WV
These project every embedding into new spaces.
Q = X × WQ K = X × WK V = X × WV
Notice that all three come from the same input embeddings.
The projections simply teach the model to represent different aspects of the same token.
Step 3: Compute similarity
Now every Query is compared against every Key.
The similarity score is simply the dot product.
score(i, j) = Qi · Kj
If two vectors point in similar directions, their score becomes larger.
For three tokens, we obtain a matrix like:
Cats chase mice Cats 2.1 1.4 0.5 chase 1.0 2.7 1.8 mice 0.6 1.5 2.3
Every row asks:
Which tokens should this token attend to?
Step 4: Why divide by √d?
Large vectors naturally produce larger dot products.
Without scaling, softmax would often become extremely “peaky,” making optimization harder.
So transformers compute
Attention(Q, K, V) = softmax(Q·Kᵀ / √dk) · V
where dk is the Key dimension.
The scaling keeps attention scores in a numerically stable range and improves training.
Step 5: Apply softmax
Raw scores are not probabilities.
Softmax converts each row into weights that sum to 1.
Example:
Raw scores [2.1, 1.4, 0.5]
becomes
Weights [0.58, 0.29, 0.13]
Now the model has a probability distribution describing how much attention each token receives.
Step 6: Mix the Values
Finally, each output becomes a weighted average of the Value vectors.
Output = weights × V
If the weights are
[0.58, 0.29, 0.13]
then the first token mostly uses information from itself, somewhat from “chase,” and a little from “mice.”
Every token repeats this process independently.
A complete runnable NumPy example
The following example intentionally keeps the dimensions tiny so every step is visible.
import numpy as np np.set_printoptions(precision=3, suppress=True) # Three toy token embeddings (3 tokens × 4 dimensions) X = np.array([ [1.0, 0.0, 1.0, 0.0], # Cats [0.0, 2.0, 0.0, 2.0], # chase [1.0, 1.0, 0.0, 0.0], # mice ]) # Simple projection matrices WQ = np.array([ [1.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 1.0], ]) WK = np.array([ [1.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 1.0], ]) WV = np.array([ [1.0, 0.0], [0.0, 1.0], [0.5, 0.0], [0.0, 0.5], ]) Q = X @ WQ K = X @ WK V = X @ WV scores = (Q @ K.T) / np.sqrt(K.shape[1]) exp_scores = np.exp(scores - scores.max(axis=1, keepdims=True)) attention = exp_scores / exp_scores.sum(axis=1, keepdims=True) output = attention @ V print("Attention weights:") print(attention) print("\nOutput vectors:") print(output)
When you run this, you’ll see:
-
every row sums to 1
-
each token attends differently
-
the output vectors become mixtures of other Value vectors
That is the heart of transformer self-attention.
The same idea in PyTorch
PyTorch makes the implementation almost identical.
import math import torch torch.manual_seed(0) X = torch.randn(4, 8) WQ = torch.randn(8, 8) WK = torch.randn(8, 8) WV = torch.randn(8, 8) Q = X @ WQ K = X @ WK V = X @ WV scores = (Q @ K.T) / math.sqrt(K.shape[-1]) weights = torch.softmax(scores, dim=-1) output = weights @ V print("Weights shape:", weights.shape) print("Output shape:", output.shape)
This closely mirrors what happens inside every transformer block, although production models use optimized fused kernels for performance.
What about multiple attention heads?
Real transformers don’t use just one attention mechanism.
They use multi-head attention.
Instead of learning one set of Q, K, and V matrices, the model learns many.
For example:
Head 1 → grammar Head 2 → long-distance references Head 3 → punctuation Head 4 → syntax ...
These aren’t hard-coded roles, but researchers consistently observe that different heads specialize in different patterns.
Each head computes attention independently.
Their outputs are concatenated and projected back into the model dimension.
This allows the model to capture several kinds of relationships simultaneously.
Why self-attention scales so well
Earlier sequence models such as RNNs process tokens one after another.
Transformers process all tokens in parallel.
Advantages include:
-
Better GPU utilization
-
Easier optimization
-
Better handling of long-range dependencies
-
Higher throughput during training
The main drawback is computational cost.
Standard self-attention compares every token with every other token.
That means computation grows roughly with the square of the sequence length.
Long-context models therefore rely on increasingly sophisticated optimizations, including memory-efficient attention kernels and architectural improvements, to make very large context windows practical.
Common beginner misconceptions
“Queries search the internet.”
No.
Queries are simply learned vectors.
They never leave the model.
“Attention replaces all previous information.”
Not exactly.
Attention produces a new representation by combining Value vectors.
Residual connections ensure earlier information is preserved and refined across layers.
“Every head learns the same thing.”
In practice, no.
Heads often develop complementary behaviors, although some become redundant.
That redundancy leads to an interesting research result.
Cherry on the cake: many attention heads are surprisingly unnecessary
One of the more surprising findings in transformer research is that many attention heads can often be removed after training with little or no measurable quality loss.
Researchers have repeatedly found that some heads contribute very little because other heads learn overlapping behavior. Modern pruning studies continue to show that carefully removing redundant heads can reduce computation and memory while preserving model quality, especially when pruning is followed by light fine-tuning.
This doesn’t mean “attention heads don’t matter.” Instead, it suggests that large transformers often learn redundant pathways, making them more robust than they first appear.
That insight has influenced ongoing work on model compression, efficient inference, and deployment on smaller hardware.
Putting everything together
A transformer block follows this simplified flow:
Input embeddings
│
▼
Linear projections
(Q, K, V)
│
▼
Q × Kᵀ
│
▼
Scale
│
▼
Softmax
│
▼
Weighted sum of V
│
▼
Output
This process repeats across many layers.
Early layers often focus on local relationships.
Middle layers build richer semantic representations.
Later layers combine information into features useful for predicting the next token.
Key takeaways
-
Self-attention lets every token examine every other token.
-
Queries ask what a token is looking for.
-
Keys describe what each token offers.
-
Values contain the information that gets combined.
-
Dot products measure similarity.
-
Scaling by √dk keeps training stable.
-
Softmax converts similarities into attention weights.
-
The final output is a weighted combination of Value vectors.
-
Multiple attention heads allow different relationships to be modeled simultaneously.
-
Modern transformers often contain redundant attention heads, enabling effective pruning without significant quality loss.
Where to go next
Understanding self-attention unlocks much of the transformer architecture, but it’s only one building block. The next concepts worth exploring are positional encodings, residual connections, layer normalization, feed-forward networks, causal masking, and efficient attention implementations for long contexts. Together, these pieces explain how modern LLMs scale from toy examples like the one above to models with billions of parameters.
If you found this guide helpful, run the NumPy example, modify the embeddings or projection matrices, and observe how the attention weights change. Experimenting with tiny examples is one of the fastest ways to build an intuition that transfers directly to real transformer models.