Transformers & Attention

The Transformer, introduced in the 2017 paper "Attention Is All You Need", completely replaced RNNs for sequence tasks and became the foundation for GPT, BERT, Gemini, and every modern large language model.

The Problem with RNNs

Before Transformers, sequences were processed with RNNs — one word at a time, left to right. This caused two big problems:

Transformers solve both problems with self-attention: every word looks at every other word simultaneously, in parallel.

Self-Attention: Every Word Talks to Every Word

Self-attention computes a relevance score between each pair of words. For the sentence "The animal didn't cross the street because it was too tired" — the word "it" needs to know it refers to "animal", not "street". Self-attention learns this.

Query, Key, Value

Each word creates three vectors: a Query (what am I looking for?), a Key (what do I contain?), and a Value (what do I give?).

import torch
import torch.nn.functional as F

def self_attention(Q, K, V):
    d_k = Q.shape[-1]
    # Attention scores: how much each token attends to others
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    weights = F.softmax(scores, dim=-1)   # Normalize to probabilities
    return torch.matmul(weights, V), weights

# In practice, PyTorch has this built in:
import torch.nn as nn
attn = nn.MultiheadAttention(embed_dim=512, num_heads=8)
output, weights = attn(query, key, value)

Multi-Head Attention

Instead of one attention mechanism, Transformers run multiple attention heads in parallel, each learning different types of relationships:

All heads' outputs are concatenated and linearly projected back to the original dimension.

Positional Encoding

Self-attention is order-agnostic — "cat sat on mat" and "mat on sat cat" would look the same. Positional encodings add position information to each token's embedding, using sine and cosine functions of different frequencies.

The Full Transformer Architecture

Why Transformers Won

Frequently Asked Questions