How Transformers Work & Deep Architecture from Scratch
A visual engineering guide to Self-Attention, Positional Encoding, and modern LLM architecture.
Deconstruct the core architectural breakthroughs powering GPT-4, Claude, and modern generative AI. Learn how multi-head attention computes context in parallel without recurrent loops.
Executive Summary & Core Thesis
Why the Transformer revolutionized artificial intelligence
Executive Summary
Before 2017, sequence processing relied on Recurrent Neural Networks (RNNs) and LSTMs that processed text word-by-word sequentially, causing severe memory bottlenecks and preventing GPU parallelization. The Transformer eliminated recurrence completely, introducing Scaled Dot-Product Self-Attention where all tokens interact in parallel.
“Attention is a communication mechanism. It treats tokens as nodes in a fully-connected directed graph where edges are dynamically weighted by relevance (similarity between Queries and Keys).”
Intellectual Journey & Timestamped Chapters
Step-by-step deconstruction aligned with the original masterclass
Follow the chronological derivation of the GPT architecture from raw characters to a working nanoGPT implementation.
Why $O(N)$ sequential steps prevented massive training scale and destroyed long-range context.
Injecting permutation awareness into order-agnostic permutation invariant attention layers.
The mathematical heart: Q*K^T / sqrt(d_k) masked with -inf to prevent future-peeking.
Allowing the model to attend to information from different representation subspaces jointly.
Unimpeded gradient flow allowing models to scale to hundreds of layers deep.
Deep Concept: The Query-Key-Value Interaction Matrix
How information is routed dynamically between tokens
Every token vector $x_i$ is linearly projected into three distinct spaces: a Query $Q_i$, a Key $K_i$, and a Value $V_i$. The dot product $Q_i \cdot K_j$ determines how much token $i$ attends to token $j$.
Scaled Dot-Product Attention Pipeline
architectureInput vectors -> Linear projections -> Scaled Dot Product -> Softmax Mask -> Weighted Value Sum
[Tokens X] ---> [ Linear Q ] -------------\
---> [ Linear K ] ---> [ MatMul Q*K^T ] ---> [ Scale 1/sqrt(d) ] ---> [ Causal Mask ] ---> [ Softmax ] ---\
---> [ Linear V ] ------------------------------------------------------------------------> [ MatMul ] ---> [ Output Z ]import torch
import torch.nn as nn
import torch.nn.functional as F
class CausalSelfAttention(nn.Module):
def __init__(self, n_embd, n_head, block_size):
super().__init__()
self.head_dim = n_embd // n_head
self.qkv_proj = nn.Linear(n_embd, 3 * n_embd, bias=False)
self.out_proj = nn.Linear(n_embd, n_embd, bias=False)
self.register_buffer("mask", torch.tril(torch.ones(block_size, block_size)))
def forward(self, x):
B, T, C = x.shape
q, k, v = self.qkv_proj(x).chunk(3, dim=-1)
# Scaled dot-product with causal mask
att = (q @ k.transpose(-2, -1)) * (1.0 / (self.head_dim ** 0.5))
att = att.masked_fill(self.mask[:T, :T] == 0, float('-inf'))
att = F.softmax(att, dim=-1)
return self.out_proj(att @ v)“Attention is fundamentally a communication mechanism: tokens vote on which other tokens have the answers they need to predict what comes next.”
Architectural Comparison: RNN / LSTM vs. Modern Transformer
Why recurrence lost the scaling war
Comparing sequential step-by-step state recurrence with parallelized attention across compute, memory, and long-range dependency horizons.
| Architectural Dimension | Classic RNN / LSTM | Transformer (Decoder-Only) |
|---|---|---|
Training Parallelization Enables trillion-token pre-training across GPU clusters. | Sequential O(N) — Token t+1 requires state from token t. | Fully Parallel O(1) across sequence dimension during training. |
Maximum Path Length Essential for maintaining coherence in 100k+ context windows. | O(N) steps — signals degrade across 100+ tokens. | O(1) direct connection — any token attends to any other token in one hop. |
Inference Complexity The one trade-off where RNN was cheaper at inference time. | O(1) constant memory and compute per new generated token. | O(N) with KV-cache / O(N^2) without KV-cache per step. |
Hardware Affinity (GPUs) Perfect match for Tensor Cores. | Memory bandwidth bound with tiny sequential matrix-vector ops. | Compute bound with massive high-throughput Matrix-Matrix (GEMM) multiplications. |
The Transformer sacrificed $O(1)$ inference cost to gain $O(1)$ path length and massive training parallelizability — unlocking the entire modern scaling era.
Step-by-Step: The Autoregressive Forward Pass
From input character prompt to next-token probability distribution
How raw string text moves through embeddings, transformer blocks, and unembedding head in 5 discrete stages.
Raw string is mapped to discrete token IDs $[t_0, t_1, \dots, t_T]$. Token embedding table $W_{tok}$ and position embedding table $W_{pos}$ are summed: $x = E_{tok} + E_{pos}$.
Tokens communicate. Queries, Keys, and Values are computed in parallel for $H$ heads. Masked softmax affinity matrix mixes information across the sequence.
Tokens think individually. Each token vector passes through an expansion layer (typically $4 \times d_{model}$), a non-linear activation (GeLU/SwiGLU), and a projection back down.
Steps 2 and 3 repeat through $L$ identical blocks (e.g. 12 layers in GPT-2 small, 96 layers in GPT-3). Information gradually ascends from syntax to semantics to logic.
Final vectors are normalized and projected via the unembedding matrix $W_{vocab}$ to produce raw logits of size $V$ (vocabulary size). Softmax with temperature generates probabilities for sampling.
Visual Evidence: Attention Patterns & Residual Highway
Inspecting what internal attention heads actually learn
Empirical captures demonstrating how attention heads specialize in specific linguistic and logical roles (e.g., induction heads, syntax parsing, delimiter tracking).
Lower Triangular Causal Mask & Attention Heatmap
Notice the intense concentration on the first token (attention sink) and recent previous tokens (recency bias), alongside specific long-distance spikes connecting pronouns to their antecedents.
Pre-LN vs Post-LN Gradient Highway
By placing LayerNorm inside the branch (Pre-LN) rather than in the main trunk (Post-LN), the identity mapping $x_{l+1} = x_l + F(x_l)$ allows gradients to backpropagate from layer 100 directly to layer 1 without degradation.
Key Insights & Engineering Gotchas
Critical heuristics from building neural networks in production
The Residual Stream is an Additive Blackboard
Think of the residual stream as a shared central memory bus. Each attention and MLP block reads from the bus, does a calculation, and adds (adds, never overwrites) its delta contribution back onto the bus.
Attention = Communication, MLP = Computation
Self-Attention is where tokens talk to each other to share context. The Feed-Forward MLP is where each token sits in isolation and processes what it has heard.
The Softmax Vanishing Gradient Trap (d_k scaling)
Without dividing by sqrt(d_k), for large vector dimensions (e.g. d_k=128), dot products grow in magnitude into the hundreds. Softmax outputs converge to one-hot vectors, yielding virtually zero gradients during backprop.
Transformer Architecture Mastery Check
Verify your understanding of Attention mechanisms and tensor dimensions.
Q1Why do we divide the Query-Key dot product by sqrt(d_k)?
Q2During causal autoregressive training, what is the role of the upper triangular mask in self-attention?
Q3What is the primary conceptual difference between Self-Attention and the Feed-Forward (MLP) layer?
Actionable Takeaways & Next Steps
What to apply immediately in your ML & AI engineering workflow
Key Synthesis Points
Attention is a dynamically weighted communication protocol over a fully connected graph.
Residual connections (Pre-LN) are essential for gradient preservation in deep networks.
Transformers achieve massive scale because training is 100% parallel matrix operations (GEMM).
Positional embeddings are mandatory because pure self-attention is permutation-invariant.
Actionable Implementation Checklist
Source Provenance & Original Attribution
Original lecture and citation metadata
Andrej Karpathy
Ex-Director of AI at Tesla, OpenAI Founding Member
Andrej Karpathy YouTube ChannelKarpathy, A. (2023). "Let's build GPT: from scratch, in code, spelled out." YouTube Masterclass Series.