-
Training instabilities: loss spikes, gradient clipping, and mixed-precision traps
A training run can look perfectly healthy for thousands of steps and then suddenly produce a loss that is 100 times larger than expected. Sometimes the model recovers. Sometimes gradients become infin