When you read a sentence, you do not pay equal attention to every word.
You look at the important ones more closely. If someone says, “The dog chased the ball because it was moving fast,” your brain instantly gives more weight to words like “dog,” “chased,” and “ball,” and less to filler words. This idea is at the heart of the attention mechanism.
Attention is a way for a model to say: “For this word, which other words matter most?”
That is the core intuition behind modern transformers.
The problem attention solves
A model sees words as numbers. These numbers are called embeddings. Each word becomes a vector in a high-dimensional space, which lets the model do math on them.
Imagine each word is a point in space. Two words that are semantically similar are closer together. But the raw embedding alone does not tell the model which words are important in context.
For example, in the sentence:
“The cat sat on the mat because it was tired.”
The word “it” refers to the cat, not the mat. A model without attention would treat all words as equally important. Attention helps the model say: “When I am reading ‘it’, I should focus on ‘cat’ more than ‘mat’.”
This is why attention is such a big idea in NLP.
A simple analogy
Think of a person in a crowded room listening to a conversation.
You do not listen equally to everyone. You pay more attention to the person who is speaking about the topic you care about. In the same way, attention makes a model focus on the most relevant tokens in a sequence.
A useful mental model is:
- each word asks, “Who should I look at?”
- each other word answers, “How relevant am I to this word?”
- the model then combines those relevant pieces into a better representation
This is the reason attention is often called a weighted lookup of context.
The toy example from the notebook
The notebook starts with a tiny set of embedding vectors. Each row is a word representation.
import torch
inputs = torch.tensor(
[[0.43, 0.15, 0.89],
[0.55, 0.87, 0.66],
[0.57, 0.85, 0.64],
[0.22, 0.58, 0.33],
[0.77, 0.25, 0.10],
[0.05, 0.80, 0.55]]
)These numbers are not meaningful by themselves. They are just a toy way to represent words in a vector space. The main point is that each token is a vector, and we can compare vectors with dot products.
Similarity: how related are two words?
The first idea is simple: compare one word vector with all the others using dot product.
query = inputs[1]
attn_scores_2 = torch.empty(inputs.shape[0])
for i, x_i in enumerate(inputs):
attn_scores_2[i] = torch.dot(x_i, query)
print(attn_scores_2)A dot product tells us how aligned two vectors are. If two vectors point in a similar direction, their dot product is larger.
This is a rough measure of similarity.
Intuition
If the current word is “journey,” then the model asks:
- How similar is this word to each other word in the sentence?
- Which ones should matter most?
That gives us a set of raw scores.
But raw scores are not yet a proper probability distribution. They can be large or small, and they do not sum to 1.
Softmax: turning scores into attention weights
This is where softmax comes in.
def softmax_naive(x):
return torch.exp(x) / torch.exp(x).sum(dim=0)
attn_weights_2_naive = softmax_naive(attn_scores_2)
print(attn_weights_2_naive)Softmax converts a list of numbers into a distribution that sums to 1. This gives us attention weights.
So instead of saying, “This score is 9.5,” we say, “This token gets 30% of the attention, that token gets 10%, and so on.”
This is a very important idea because attention is not just a ranking. It is a weighted decision.
Why softmax matters
Without softmax, the model cannot easily compare the scores as meaningful probabilities. Softmax makes the scores usable as attention weights.
It is like saying:
- I care about these three words more than the rest
- here is the proportion of my focus for each one
Creating a context vector
Once we have attention weights, we can combine the word vectors into one weighted representation.
context_vec_2 = torch.zeros(query.shape)
for i, x_i in enumerate(inputs):
context_vec_2 += attn_weights_2[i] * x_i
print(context_vec_2)This new vector is a context-aware representation of the current word.
In plain English, it means:
The model has looked at all words, decided which ones matter, and blended their information into a better encoding of the current word.
This is much richer than using the original word vector alone.
Why this is powerful
The model is no longer stuck with a single static embedding for a word. It can make that embedding depend on context.
The same word can mean different things in different contexts.
- “bank” can mean a financial institution
- “bank” can also mean a river edge
Attention helps the model decide which meaning is relevant based on surrounding words.
The word is not treated as isolated. It is interpreted in context.
The full attention matrix
The notebook then expands the idea from one word to the entire sequence.
Instead of one query, we compute attention between every pair of tokens.
attn_scores = inputs @ inputs.T
print(attn_scores)This creates a matrix. Each row tells us how much a token should pay attention to every other token in the sequence.
That is the heart of self-attention:
- every token looks at every other token
- it computes relevance scores
- it aggregates information from the most relevant ones
Learnable attention: the model learns what matters
The earlier examples used fixed dot products between embeddings. That is a nice intuition, but real models do something better: they learn the attention weights.
They do this by creating projection matrices for:
- queries
- keys
- values
W_query = torch.nn.Parameter(torch.rand(d_in, d_out))
W_key = torch.nn.Parameter(torch.rand(d_in, d_out))
W_value = torch.nn.Parameter(torch.rand(d_in, d_out))These matrices are learned during training. This means the model does not just compute similarity using raw embeddings. It learns the best way to transform each token before comparing them.
Query, key, value
This is a classic transformer idea.
- Query: what am I looking for?
- Key: what information do I have to offer?
- Value: the actual content I can pass along
A token asks, “Which other tokens are relevant to me?” It compares its query with all keys. Then it gathers the corresponding value vectors.
This is a very clean and powerful abstraction.
Why do we scale by sqrt(d_k)?
The notebook divides scores by the square root of the key dimension.
attn_weights_2 = torch.softmax(attn_scores_2 / d_k**0.5, dim=-1)This is a stability trick.
If the dimensions get large, the dot product values can become large and softmax may become too sharp. Scaling keeps the distribution stable.
So even though this is mathematically small, it is very important in practice.
Self-attention in real terms
Self-attention means each token attends to other tokens in the same sequence.
This is what allows a model to understand relationships inside a sentence.
For example, in the sentence:
“The book that the teacher recommended was helpful.”
The model can learn that “book” is the thing being discussed, while “teacher” and “recommended” are relevant context words.
This is what makes attention so useful in language understanding.
Causal attention and masking
One special type of attention is causal attention.
In language generation, a token should only look at words that come before it, not future words.
Otherwise, when predicting the next word, the model would be cheating by looking ahead.
The notebook shows this with a mask.
mask = torch.triu(torch.ones(context_length, context_length), diagonal=1)
masked = attn_scores.masked_fill(mask.bool(), -torch.inf)
attn_weights = torch.softmax(masked / keys.shape[-1]**0.5, dim=1)A causal mask blocks future positions. This is why models like GPT can generate text one token at a time.
Intuition
It is like reading a sentence word by word:
- you can only use what is already visible
- you cannot peek at the next word
- you must decide based on context so far
This is how autoregressive language models work.
Multi-head attention
A single attention mechanism may learn one kind of relationship, but not all of them.
That is why transformers use multi-head attention.
Instead of one attention process, they run several in parallel.
Each head can focus on a different kind of pattern:
- one head may focus on noun phrases
- one on pronoun references
- one on grammatical structure
- another on semantic similarity
This is the same reason a team can do better than one person alone. Different heads specialize in different views of the same sentence.
The notebook shows a small version of this idea:
class MultiHeadAttention(nn.Module):
def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):
...The key point is not the exact code. The key point is the idea:
multiple attention heads let the model look at the same sentence from different angles.
Dropout
The notebook also includes dropout.
This is a regularization trick used in deep learning.
dropout = torch.nn.Dropout(0.5)
print(dropout(example))Dropout randomly switches some values off during training. This stops the model from relying too heavily on one path and helps it generalize better.
Intuition
It is like a student practicing with occasional distractions. The student cannot memorize one exact pattern; instead, it learns more robust ideas.
Why attention changed everything
Before attention, models struggled with long-range dependencies.
If a model had to understand a sentence like:
“The book that the teacher recommended last week was extremely useful and it changed my perspective.”
the model needed a way to keep track of relevant context far away from the current token.
Attention solves this elegantly by letting each token directly look at the tokens it needs.
This is one of the main reasons transformers became the backbone of modern NLP.
Final intuition
If I had to reduce the entire idea to one sentence, it would be this:
Attention is a way for a model to decide, for each word, which other words are worth paying attention to.
It is not magic. It is just a structured way to weigh context.
And once you understand that idea, the rest of transformer architecture starts to make much more sense.
Reference
This post is based on the attention notebook in GitHub: attention_mechanism.ipynb, which follows the ideas from the book Natural Language Processing from Scratch.
If you want the next step, the natural follow-up is to read about transformers and multi-head attention in a more end-to-end setting. That is where all of these ideas become the engine behind GPT, BERT, and modern language models.