-
Training instabilities: loss spikes, gradient clipping, and mixed-precision traps
A training run can look perfectly healthy for thousands of steps and then suddenly produce a loss that is 100 times larger than expected. Sometimes the model recovers. Sometimes gradients become infin
-
Weight initialization: why your deep net trains or dies before step one
A deep network does not begin learning from a neutral state. Before the optimizer takes its first step, the initial weights have already determined: