Intro to Large Language Models: Pre-training, RLHF & LLM OS
A foundational masterclass on how base models are trained, aligned into assistants, and scaled into the kernel of a new operating system.
Andrej Karpathy's comprehensive breakdown of generative AI: from lossy internet compression on 10TB of text to RLHF alignment, System 2 tree search, and the LLM OS computing paradigm.
Executive Summary & The Dual-Stage Paradigm
The fundamental dichotomy between Pre-training and Post-training
Executive Summary
Large Language Models are not databases; they are lossy, probabilistic compression files of human knowledge. Building a modern AI assistant requires two completely distinct stages: 1) Pre-training (~$2M-$100M compute on 10TB+ text) to produce a raw Base Model, and 2) Post-training (SFT + RLHF on high-quality conversations) to align the model into a polite, tool-using assistant.
“The LLM is emerging as the central processing unit (CPU/kernel) of a new computing architecture, where the context window functions as RAM and external APIs/databases function as disk storage.”
Intellectual Journey & Timestamped Chapters
7 core milestones from raw tokens to the operating system paradigm
Chronological breakdown of Karpathy's 1-hour masterclass with direct links to the source lecture.
An LLM is just two files: a ~140GB parameters file (weights) and a ~500-line C file (llama2.c) that runs matrix multiplication on CPU/GPU.
Scraping ~10TB of text, tokenizing into discrete integers, and running thousands of GPUs for months to predict next tokens.
Why raw base models answer questions with more questions, and how human contractors write Q&A dialogues to align models into assistants.
Next-token prediction loss scales smoothly and predictably with compute (FLOPs) and dataset size.
LLMs emit special control tokens (e.g. |BROWSER|) to execute Python scripts, calculate arithmetic, and fetch live web data.
Current LLMs are pure System 1 (instinctive, constant time per token). Future frontier models will search reasoning trees (System 2 deliberate reflection).
The LLM as the CPU of a new operating system, coordinating RAM (context window), disk (files), and peripherals, alongside emerging security threats like prompt injection.
Core Concept: The LLM as the Operating System Kernel
How the computer architecture mental model perfectly maps to modern AI systems
Just as a CPU executes assembly instructions and coordinates RAM, disk storage, and peripherals, an LLM orchestrates natural language tokens across its working context window (RAM), vector stores / files (Disk), and external APIs / browsers (Peripherals).
The LLM Operating System Architecture
architectureComparison of traditional OS architecture with LLM OS stack.
================= THE LLM OPERATING SYSTEM =================
[ USER / APPLICATIONS ] ---> ( NATURAL LANGUAGE INTENT )
|
v
+-----------------------------------------+
| LLM KERNEL / CPU |
| (Transformer Neural Network) |
+-----------------------------------------+
| | |
v v v
[ CONTEXT WINDOW ] [ DISK / RAG ] [ PERIPHERALS ]
( RAM Memory ) (Vector Stores) (Tools & APIs)
- Working instructions - PDF documents - Python Shell
- Intermediate chats - Code repos - Web Browser
- Prompt tokens - DB records - Calculator
============================================================“An LLM is not a chatbot; it is the central processing unit of a new computing architecture that communicates in natural language.”
The 3-Phase Pipeline: From Raw Internet to AI Assistant
How billions of unstructured web tokens become polite, helpful assistants
The exact three developmental stages required to create production models like ChatGPT, Claude, and Llama.
Scrape ~10TB of raw text from Common Crawl, Wikipedia, GitHub, and books. Train thousands of GPUs for months to minimize next-token cross-entropy loss. Output: a base model (e.g. Llama 3 70B Base) that acts as an internet document completer.
Human contractors write ~100,000 high-quality Q&A prompt-response pairs. Finetune the base model to swap document completion behavior for conversational assistant behavior.
Human evaluators rank multiple model completions from best to worst. Train a reward model and optimize the LLM policy (via PPO or DPO) to maximize helpfulness, accuracy, and safety.
Comparative Analysis: Base Model vs. Aligned Assistant
Why using a raw base model for chat fails and how alignment shifts behavior
Contrasting raw pre-trained base models with fine-tuned conversational assistants across behavior, prompt response, and reliability.
| Dimension | Base Model (Pre-trained) | Assistant Model (SFT + RLHF) |
|---|---|---|
Core Objective | Complete the document statistically based on internet distribution. | Answer questions helpfully, politely, and truthfully as an assistant. |
Response to a Question | Often responds with more questions (imitating an online homework sheet). | Provides direct, concise, formatted answers. |
Training Cost Pre-training acquires all world knowledge; post-training merely unlocks it. | 99% of total compute budget ($10M - $100M+ in GPU time). | 1% of total compute budget (cheap finetuning on curated data). |
Hallucination Nature | Wild daydreaming across all internet tropes. | Directed dreams constrained into helpful assistant tone. |
Post-training does not teach the model new facts—it teaches the model how to converse, format answers, and access its existing latent knowledge.
Visual Evidence: The LLM OS & Pre-training Loss Scaling
Inspecting the empirical mechanics of the new computing stack
Empirical slides and architectural diagrams from Karpathy's masterclass comparing computer architectures and scaling laws.
The LLM Operating System Stack Diagram
The central processing unit is the Transformer neural network. The context window acts as limited, volatile RAM, while disk storage is semantic retrieval over external databases.
Pre-training Test Loss Smooth Power-Law Scaling
Unlike downstream qualitative benchmarks which jump erratically, next-token prediction loss follows exact power laws as compute and dataset volume increase.
Mental Models & AI Security Principles
Key heuristics from Andrej Karpathy's analysis
Hallucination is the Default State of an LLM
An LLM is a creative, probabilistic simulator. It only ever dreams text. When the dream happens to align with physical reality, we call it knowledge. When it diverges, we call it hallucination.
Prompt Injection is the SQL Injection of LLMs
Because LLMs process instructions and data in the exact same input stream, malicious text inside a processed PDF or webpage can hijack the control flow of the model.
System 1 vs System 2 Thinking Gap
Humans spend hours deliberating before making hard decisions (System 2). LLMs currently spend the exact same number of FLOPs on each token regardless of problem difficulty. Future breakthroughs will come from reasoning tree search.
Large Language Models Assessment
Verify your understanding of the foundational concepts explained by Andrej Karpathy.
Q1What is the primary role of Supervised Finetuning (SFT) in creating an AI assistant?
Q2In the 'LLM as an Operating System' analogy, what component corresponds to computer RAM?
Q3Why is Prompt Injection fundamentally difficult to eliminate in LLM applications?
Actionable Takeaways & AI Strategy
Key recommendations for software developers and AI practitioners
Key Synthesis Points
Pre-training creates knowledge; SFT and RLHF create alignment and formatting.
Treat the LLM as a CPU orchestrator and context window as limited, precious RAM.
Mitigate hallucinations with retrieval-augmented generation and tool execution.
Design for prompt injection vulnerabilities from day one.
Actionable Implementation Checklist
Source Provenance & Attribution
Direct attribution to the original lecture and presentation materials
Andrej Karpathy
Ex-Director of AI at Tesla, OpenAI Founding Member
Andrej Karpathy YouTube ChannelKarpathy, A. (2023). "[1hr Talk] Intro to Large Language Models." YouTube Educational Talk.