-
Training instabilities: loss spikes, gradient clipping, and mixed-precision traps
A training run can look perfectly healthy for thousands of steps and then suddenly produce a loss that is 100 times larger than expected. Sometimes the model recovers. Sometimes gradients become infin
-
Recurrent nets to attention: why sequence models dropped the loop
For years, the standard approach to understanding a sequence was straightforward: read one element, update an internal memory, move to the next element, and repeat.
-
Convolutions from scratch: what a CNN actually sees, layer by layer
A convolutional neural network does not look at an image the way a human does. It does not immediately recognize a bicycle, a building, or a handwritten digit.
-
Overfitting on purpose: dropout, weight decay, and early stopping in one runnable lab
Machine learning tutorials often introduce overfitting as a failure mode: the model memorizes the training data, performs brilliantly on examples it has already seen, and then collapses on new data.
-
Optimizers beyond SGD: Adam, AdamW, and the learning-rate schedule that matters more
Training a neural network is an optimization problem disguised as an engineering project. We choose an architecture, prepare data, define a loss function, and then repeatedly update millions of parame
-
Batch norm, layer norm, RMSNorm: what normalization actually fixes
Neural-network normalization is often taught as a simple recipe: subtract a mean, divide by a standard deviation, and training gets easier.
-
Weight initialization: why your deep net trains or dies before step one
A deep network does not begin learning from a neutral state. Before the optimizer takes its first step, the initial weights have already determined:
-
Activation functions in 2026: why ReLU won and what’s replacing it
ReLU remains the cheapest dependable baseline, but modern transformer feed-forward blocks increasingly favor smooth, gated designs—especially GELU, SiLU, and SwiGLU—because they preserve a gradient pa
-
Backpropagation demystified: hand-compute the gradients, then verify with autograd
Backpropagation is the engine behind modern deep learning. Whether you’re training a tiny multilayer perceptron or a frontier-scale language model, every optimization step depends on efficiently compu
-
A neuron is just a weighted sum: build a perceptron in 20 lines of NumPy
Modern AI systems can contain billions of parameters, but the core idea behind a neural network starts with a much smaller building block.