How Does One Short Sentence Change the Model?
Follow one batch through an update, then inspect how the same network generates text after 1,000 pretraining updates and 300 question-answer fine-tuning updates.
Random initialization→Text pretraining × 1,000→Question-answer fine-tuning × 300