HELLO LLM · FREE TO LEARN · NONCOMMERCIAL
Build a small model.
Understand large models.
Start with a neuron you can calculate by hand.
Build a small language model that learns and generates text.
Start from zero · 20 lessons · No GPU needed
Put the ideas to work
Change a value, edit code, then train a model yourself.
A neuron, step by step
Change inputs and parameters. Check each step of a neuron's calculation.
Open experiment3D model inspector
Follow real tensors through the model's forward pass.
Open experimentTrain the complete model
Train it yourself and compare the loss curves and generated text.
Open experimentShow the other 14 experiments
Video lessons
From principles to source code and results.
Build the model, step by step
One project throughout. A result you can check at every stage.
Start here
Meet the course, then put the finished model to the test.
Make a numerical network learn
Start with one output. See how errors change parameters.
- 01Neurons: What Does a Model Calculate Before It Writes the Next Character?Explain the roles of inputs, parameters, and activation.
- 02Vectors and Matrices: Do Two Outputs Require Two Copies of the Code?Explain input, parameter, and output shapes axis by axis.
- 03Nonlinearity: Why Can Multiply-and-Add Not Express Four Simple Rules?Explain the difference between stacking layers and adding nonlinearity.
- 04Gradients: Why Can an Update in the Right Direction Make Things Worse?Distinguish the local direction given by a gradient from the actual update step size.
- 05Backpropagation: Must We Test All 9 Parameters One by One?Calculate one backward path by hand and explain how branch contributions combine before parameters are updated together.
- 06The Training Loop: What Happens to Four Predictions After 500 Updates?Write one training iteration, explain gradient clearing, and identify incorrect comparisons caused by broadcasting.
Stage goal: Complete your first training loop
Bring text into the network
Turn text into inputs, probabilities, and training examples.
- 07Embedding: How Does Text Become a Network Input?Distinguish character IDs, embedding vectors, and the three tensor axes.
- 08Softmax: How Do Scores Become Next-Character Probabilities?Explain candidate scores, relative competition, and normalization.
- 09Training Targets: How Does a Sentence Become a Learnable Task?Construct shifted targets and explain average loss over valid positions.
Stage goal: Train your first character prediction model
Read context and assemble the model
Add context and order, then connect the components.
- 10Attention: How Does Predicting the Next Character Read Earlier Text?Calculate one attention row and distinguish reading proportions from the content being read.
- 11Causal Masks: Why Is Parallel Training Not Peeking at the Answer?Explain causal visibility and the correct placement of the mask.
- 12Multi-Head Attention: Where Does Each Head Read?Explain head splitting, independent attention weights, and merging, and identify incorrect axes even when shapes match.
- 13RoPE: How Does the Model Know the Order?Connect the two-dimensional rotation formula to relative positions and Q/K dot products.
- 14SwiGLU: What Happens After Reading the Prefix?Explain input-dependent gating and positionwise processing.
- 15Residual Connections and RMSNorm: Why Keep a Direct Path?Assemble head splitting, rotation, and causal aggregation into an attention component, then add residual connections and a Pre-Norm model block.
- 16The Complete Model: Follow the Character “欢” Through Every StepAssemble the complete model and distinguish representations, parameters, and vocabulary scores.
Stage goal: Trace a forward pass through the complete model
Pretrain, generate, and fine-tune
Teach the same model to continue text, then answer questions.
- 17Pretraining: Why Can New Questions Get Worse as Training Questions Improve?Complete pretraining and use differences in the data to explain changes in evaluation results.
- 18Generation: How Many Choices Does One Sentence Take?Write a generation loop and explain random selection and stopping conditions.
- 19Supervised Fine-Tuning: Why Can It Continue Text but Still Fail to Answer?Build answer supervision and diagnose fine-tuning tradeoffs with fixed comparisons across three methods and three seeds.
- 20The Full Workflow: Connect Pretraining, Fine-Tuning, and GenerationIndependently complete a transfer experiment covering data, vocabulary, model construction, training, and reloading, and explain both successes and failures.
Stage goal: Reproduce training and test your understanding on new data
After the 20 lessons
Return to the beginning and see how much you can now explain.
Download the book
Use PDF for printing, or EPUB for phones and e-readers.
Chinese · v0.1.0-rc.1
