-
Optimizers beyond SGD: Adam, AdamW, and the learning-rate schedule that matters more
Training a neural network is an optimization problem disguised as an engineering project. We choose an architecture, prepare data, define a loss function, and then repeatedly update millions of parame
-
Batch norm, layer norm, RMSNorm: what normalization actually fixes
Neural-network normalization is often taught as a simple recipe: subtract a mean, divide by a standard deviation, and training gets easier.
-
Weight initialization: why your deep net trains or dies before step one
A deep network does not begin learning from a neutral state. Before the optimizer takes its first step, the initial weights have already determined:
-
Activation functions in 2026: why ReLU won and what’s replacing it
ReLU remains the cheapest dependable baseline, but modern transformer feed-forward blocks increasingly favor smooth, gated designs—especially GELU, SiLU, and SwiGLU—because they preserve a gradient pa
-
Backpropagation demystified: hand-compute the gradients, then verify with autograd
Backpropagation is the engine behind modern deep learning. Whether you’re training a tiny multilayer perceptron or a frontier-scale language model, every optimization step depends on efficiently compu
-
A neuron is just a weighted sum: build a perceptron in 20 lines of NumPy
Modern AI systems can contain billions of parameters, but the core idea behind a neural network starts with a much smaller building block.