Derivatives and gradients: the one calculus lesson ML actually uses

LEARN · MATHEMATICS FOR MACHINE LEARNING

Machine learning can look like a huge collection of algorithms, architectures, and optimization tricks. Underneath most of them, however, sits one compact idea:

Measure how a small parameter change affects the loss, then move the parameter in the direction that reduces the loss.

The derivative measures that effect for one variable. A gradient collects the derivatives for many variables. Gradient descent turns those measurements into an iterative learning algorithm.

This lesson develops those ideas from the ground up, connects them to model training, and makes them visible on two-dimensional loss surfaces. By the end, you will have runnable Python scripts for numerical derivatives, linear regression, contour plots, gradient checking, and an animated optimizer navigating a deceptive spiral-shaped valley.

What you will learn

You will learn how to:

Interpret a derivative as a local rate of change.
Approximate derivatives numerically.
Read a gradient as a direction in parameter space.
Derive the gradient-descent update rule.
Apply the chain rule to a machine-learning loss.
Train a linear model without a machine-learning framework.
Plot an optimizer’s path across a two-dimensional loss surface.
Diagnose learning rates that are too small or too large.
Check an analytical gradient with finite differences.
Understand why curved and narrow valleys can confuse optimizers.

Set up the runnable environment

Create an isolated Python environment:

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Activate it on Windows PowerShell:

.venv\Scripts\Activate.ps1

Install the packages used in the examples:

python -m pip install --upgrade pip
python -m pip install numpy matplotlib pillow

Python’s current documentation continues to recommend the built-in venv module for creating isolated environments. The animation examples below use Matplotlib’s documented FuncAnimation and PillowWriter APIs.

A derivative answers a local question

Suppose a model has one parameter, w, and its loss is:

loss(w) = (w - 3)² + 1

This function reaches its minimum at w = 3. Its derivative is:

d loss / d w = 2(w - 3)

The derivative tells you how the loss changes when w changes by a very small amount.

At w = -4:

d loss / d w = 2(-4 - 3) = -14

The negative sign means increasing w should reduce the loss.

At w = 7:

d loss / d w = 2(7 - 3) = 8

The positive sign means increasing w would raise the loss, so decreasing w should help.

At w = 3:

d loss / d w = 0

A zero derivative means there is no first-order downhill direction. The point may be a minimum, maximum, saddle point, or flat region. In this simple bowl-shaped function, it is the minimum.

The derivative is not the function value

This distinction matters:

The function value tells you how high the loss is.
The derivative tells you which way the loss is changing.
A large loss can have a small derivative.
A small loss can have a large derivative.
A derivative of zero does not automatically prove that you found the global minimum.

An optimizer usually needs both pieces of information: the loss for monitoring and the derivative for deciding how to update the parameters.

Estimating a derivative numerically

You do not always need symbolic calculus to estimate a derivative. You can evaluate the function slightly to the left and slightly to the right.

The central-difference approximation is:

f′(x) ≈ [f(x + h) - f(x - h)] / (2h)

Here, h is a small positive step.

Create finite_difference.py:

def loss(w: float) -> float:
    return (w - 3.0) ** 2 + 1.0


def analytical_derivative(w: float) -> float:
    return 2.0 * (w - 3.0)


def numerical_derivative(w: float, h: float = 1e-5) -> float:
    return (loss(w + h) - loss(w - h)) / (2.0 * h)


def main() -> None:
    for w in [-4.0, 0.0, 3.0, 7.0]:
        exact = analytical_derivative(w)
        estimated = numerical_derivative(w)
        print(
            f"w={w:5.1f} "
            f"analytical={exact:10.6f} "
            f"numerical={estimated:10.6f}"
        )


if __name__ == "__main__":
    main()

Run it:

python finite_difference.py

You should see the numerical and analytical answers agree closely:

w= -4.0 analytical=-14.000000 numerical=-14.000000
w=  0.0 analytical= -6.000000 numerical= -6.000000
w=  3.0 analytical=  0.000000 numerical=  0.000000
w=  7.0 analytical=  8.000000 numerical=  8.000000

Central differences generally provide a more accurate local estimate than a one-sided difference using the same step size. NumPy’s current gradient implementation likewise uses central differences for interior samples and one-sided differences at boundaries.

Why not make `h` infinitely small?

Smaller is not always better.

If h is too large, the approximation examines points that are too far from x, so it may miss the truly local slope.

If h is too small, floating-point subtraction can lose precision because f(x + h) and f(x - h) become nearly identical numbers.

In everyday double-precision calculations, values around 1e-4 to 1e-6 are often useful starting points for gradient checking. The best value depends on the function’s scale and numerical behavior.

From derivatives to partial derivatives

Real models have more than one parameter.

Imagine a loss depending on two parameters:

L(x, y) = (x - 1)² + 4(y + 0.5)² + 0.5xy

To understand how x affects the loss, treat y as fixed and differentiate with respect to x:

∂L/∂x = 2(x - 1) + 0.5y

To understand how y affects the loss, treat x as fixed:

∂L/∂y = 8(y + 0.5) + 0.5x

These are partial derivatives. Each one answers a local question about one coordinate while holding the others constant.

A gradient packages the partial derivatives

The gradient is the vector of partial derivatives:

∇L(x, y) = [∂L/∂x, ∂L/∂y]

For the example above:

∇L(x, y) = [2(x - 1) + 0.5y, 8(y + 0.5) + 0.5x]

At the point (x, y) = (3, 2):

∂L/∂x = 2(3 - 1) + 0.5(2) = 5
∂L/∂y = 8(2 + 0.5) + 0.5(3) = 21.5

Therefore:

∇L(3, 2) = [5, 21.5]

The gradient points in the direction of the steepest local increase. Its negative points in the direction of the steepest local decrease.

That gives us an optimization rule:

new parameters = old parameters - learning rate × gradient

Using the common symbol η for the learning rate:

θ_next = θ_current - η∇L(θ_current)

The minus sign is the whole strategy: the gradient points uphill, so subtracting it moves downhill.

Gradient descent in one dimension

Return to:

loss(w) = (w - 3)² + 1

Start at w = -4 and use a learning rate of 0.2.

Create one_dimensional_descent.py:

def loss(w: float) -> float:
    return (w - 3.0) ** 2 + 1.0


def gradient(w: float) -> float:
    return 2.0 * (w - 3.0)


def main() -> None:
    w = -4.0
    learning_rate = 0.2

    for step in range(8):
        current_loss = loss(w)
        current_gradient = gradient(w)

        print(
            f"{step:02d} "
            f"w={w: .6f} "
            f"loss={current_loss: .6f} "
            f"gradient={current_gradient: .6f}"
        )

        w = w - learning_rate * current_gradient


if __name__ == "__main__":
    main()

Run it:

python one_dimensional_descent.py

The output is:

00 w=-4.000000 loss= 50.000000 gradient=-14.000000
01 w=-1.200000 loss= 18.640000 gradient=-8.400000
02 w= 0.480000 loss= 7.350400 gradient=-5.040000
03 w= 1.488000 loss= 3.286144 gradient=-3.024000
04 w= 2.092800 loss= 1.823012 gradient=-1.814400
05 w= 2.455680 loss= 1.296284 gradient=-1.088640
06 w= 2.673408 loss= 1.106662 gradient=-0.653184
07 w= 2.804045 loss= 1.038398 gradient=-0.391910

Notice the pattern:

The parameter begins far from the minimum.
Its gradient is large and negative.
Subtracting a negative number moves w to the right.
As w approaches 3, the gradient gets smaller.
The updates shrink naturally near the minimum.

The learning rate scales every update. The gradient decides the direction and local sensitivity; the learning rate decides how aggressively to respond.

The chain rule is the part machine learning really needs

A model’s loss is rarely a direct function of one parameter. It is usually a sequence of nested calculations.

Consider one training example:

prediction = w × x + b
error = prediction - target
loss = error²

To find how w affects the loss, trace the dependency chain:

w → prediction → error → loss

The chain rule multiplies the local derivatives along that path.

The local pieces are:

d loss / d error = 2 × error
d error / d prediction = 1
d prediction / d w = x

Multiply them:

d loss / d w = 2 × error × x

For the bias:

d prediction / d b = 1

Therefore:

d loss / d b = 2 × error

That is backpropagation in miniature. A neural network contains far more operations and parameters, but the underlying process is still repeated application of the chain rule.

Training a linear model from scratch

Let us fit a simple relationship between server request load and response latency.

The dataset is synthetic, reproducible, and contains no sensitive or medical information.

The model is:

predicted latency = slope × requests per second + intercept

We will minimize mean squared error:

MSE = average((prediction - observed latency)²)

Create manual_linear_regression.py:

import numpy as np


def main() -> None:
    rng = np.random.default_rng(7)

    requests_per_second = rng.uniform(
        low=50.0,
        high=900.0,
        size=500,
    )

    latency_ms = (
        18.0
        + 0.42 * requests_per_second
        + rng.normal(
            loc=0.0,
            scale=25.0,
            size=requests_per_second.size,
        )
    )

    feature_mean = requests_per_second.mean()
    feature_scale = requests_per_second.std()
    x = (requests_per_second - feature_mean) / feature_scale
    y = latency_ms

    weight = 0.0
    bias = y.mean()
    learning_rate = 0.05
    steps = 1_000

    for step in range(steps):
        predictions = weight * x + bias
        errors = predictions - y
        loss = np.mean(errors**2)

        weight_gradient = 2.0 * np.mean(errors * x)
        bias_gradient = 2.0 * np.mean(errors)

        weight -= learning_rate * weight_gradient
        bias -= learning_rate * bias_gradient

        if step % 100 == 0 or step == steps - 1:
            print(
                f"step={step:04d} "
                f"loss={loss:10.4f} "
                f"weight={weight:10.4f} "
                f"bias={bias:10.4f}"
            )

    slope_per_request = weight / feature_scale
    physical_intercept = (
        bias - weight * feature_mean / feature_scale
    )

    print()
    print(f"Estimated slope: {slope_per_request:.4f} ms per request/s")
    print(f"Estimated intercept: {physical_intercept:.4f} ms")


if __name__ == "__main__":
    main()

Run it:

python manual_linear_regression.py

The estimated slope should be close to the synthetic data generator’s true value of 0.42.

Where the batch gradient comes from

For one example:

d loss / d weight = 2 × error × x

For a batch, average those per-example derivatives:

weight gradient = 2 × average(error × x)

Similarly:

bias gradient = 2 × average(error)

The code translates those equations directly:

weight_gradient = 2.0 * np.mean(errors * x)
bias_gradient = 2.0 * np.mean(errors)

The gradient is not an abstract extra object added by the optimizer. It is the accumulated sensitivity of the loss to the model’s parameters.

Why standardize the input?

The raw feature ranges from roughly 50 to 900. Standardizing it produces a feature centered near zero with a standard deviation near one.

This improves the geometry of the optimization problem. When parameters operate on dramatically different numerical scales, the loss surface can become stretched into a long, narrow valley. A single learning rate then struggles to make sensible progress in every direction.

Standardization does not change the basic linear relationship. It changes the coordinate system in which optimization happens.

Visualizing descent on a two-dimensional loss surface

A contour plot turns a three-dimensional surface into a top-down map.

Each contour line joins points with the same loss:

Wide spacing means the surface changes slowly.
Tight spacing means the surface changes rapidly.
Nested contours often surround a minimum.
An elongated contour pattern indicates different curvature in different directions.

Create quadratic_contour.py:

from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np


def loss(points: np.ndarray) -> np.ndarray:
    points = np.asarray(points, dtype=float)
    x = points[..., 0]
    y = points[..., 1]

    return (
        (x - 1.0) ** 2
        + 4.0 * (y + 0.5) ** 2
        + 0.5 * x * y
    )


def gradient(point: np.ndarray) -> np.ndarray:
    x, y = np.asarray(point, dtype=float)

    return np.array(
        [
            2.0 * (x - 1.0) + 0.5 * y,
            8.0 * (y + 0.5) + 0.5 * x,
        ]
    )


def run_gradient_descent(
    start: tuple[float, float],
    learning_rate: float,
    steps: int,
) -> np.ndarray:
    path = [np.asarray(start, dtype=float)]

    for _ in range(steps):
        next_point = path[-1] - learning_rate * gradient(path[-1])
        path.append(next_point)

    return np.vstack(path)


def main() -> None:
    x_values = np.linspace(-4.0, 5.0, 300)
    y_values = np.linspace(-3.0, 3.0, 300)
    x_grid, y_grid = np.meshgrid(x_values, y_values)

    grid_points = np.stack([x_grid, y_grid], axis=-1)
    loss_grid = loss(grid_points)

    path = run_gradient_descent(
        start=(-3.5, 2.5),
        learning_rate=0.12,
        steps=35,
    )

    figure, axis = plt.subplots(figsize=(9, 7))

    contours = axis.contour(
        x_grid,
        y_grid,
        loss_grid,
        levels=28,
    )
    axis.clabel(contours, inline=True, fontsize=7)

    axis.plot(
        path[:, 0],
        path[:, 1],
        marker="o",
        markersize=3,
        linewidth=1.5,
        label="Gradient-descent path",
    )

    axis.scatter(
        path[0, 0],
        path[0, 1],
        s=90,
        marker="s",
        label="Start",
    )

    axis.scatter(
        path[-1, 0],
        path[-1, 1],
        s=90,
        marker="*",
        label="Final point",
    )

    axis.set_title("Gradient descent on a two-parameter loss")
    axis.set_xlabel("Parameter x")
    axis.set_ylabel("Parameter y")
    axis.legend()
    axis.set_aspect("equal", adjustable="box")

    output_path = Path("quadratic_descent.png")
    figure.tight_layout()
    figure.savefig(output_path, dpi=160)
    plt.show()

    print(f"Saved {output_path.resolve()}")
    print(f"Final point: {path[-1]}")
    print(f"Final loss: {loss(path[-1]):.8f}")


if __name__ == "__main__":
    main()

Run it:

python quadratic_contour.py

The optimizer does not usually travel in a perfectly straight line. The surface is steeper in the y direction than in the x direction, so the gradient reacts more strongly to vertical displacement.

This produces the characteristic zig-zag pattern seen in narrow valleys: a step crosses the valley, the next gradient points back across it, and useful progress along the valley happens more slowly.

What the learning rate really controls

The learning rate is often described as the “step size,” but that description is incomplete.

The actual update is:

update = learning rate × gradient

A learning rate of 0.1 does not mean the parameter moves exactly 0.1. A gradient of magnitude 50 would produce an update of magnitude 5, while a gradient of magnitude 0.001 would produce an update of magnitude 0.0001.

When the learning rate is too small

Typical symptoms include:

The loss decreases, but extremely slowly.
Training appears stable but makes little progress.
Parameters move only tiny distances.
A fixed training budget ends before convergence.

When the learning rate is too large

Typical symptoms include:

The loss jumps up and down.
The optimizer repeatedly crosses a narrow valley.
Parameter values grow rapidly.
The loss becomes inf or nan.
Training briefly improves and then diverges.

When the learning rate is reasonable

You often see:

Rapid improvement early in training.
Smaller effective updates near a smooth minimum.
Mostly downward loss movement for full-batch descent.
Stable parameter values.
A final plateau caused by convergence, noise, or limited model capacity.

For mini-batch training, individual loss measurements can fluctuate even when the overall trend is improving. The sampled batch changes from one update to the next, so the estimated gradient contains noise.

Gradient checking catches implementation mistakes

Hand-derived gradients are easy to get subtly wrong.

Common bugs include:

Missing a factor of 2.
Using addition instead of subtraction.
Averaging over the wrong axis.
Forgetting a chain-rule factor.
Broadcasting an array into an unintended shape.
Differentiating the regularization term incorrectly.

Gradient checking compares an analytical gradient with a numerical central-difference estimate.

Create gradient_check.py:

import numpy as np


def loss(parameters: np.ndarray) -> float:
    x, y = parameters

    return float(
        (x - 1.0) ** 2
        + 4.0 * (y + 0.5) ** 2
        + 0.5 * x * y
    )


def analytical_gradient(parameters: np.ndarray) -> np.ndarray:
    x, y = parameters

    return np.array(
        [
            2.0 * (x - 1.0) + 0.5 * y,
            8.0 * (y + 0.5) + 0.5 * x,
        ]
    )


def numerical_gradient(
    function,
    parameters: np.ndarray,
    step_size: float = 1e-5,
) -> np.ndarray:
    parameters = np.asarray(parameters, dtype=float)
    result = np.zeros_like(parameters)

    for index in range(parameters.size):
        offset = np.zeros_like(parameters)
        offset[index] = step_size

        right_value = function(parameters + offset)
        left_value = function(parameters - offset)

        result[index] = (
            right_value - left_value
        ) / (2.0 * step_size)

    return result


def relative_error(
    analytical: np.ndarray,
    numerical: np.ndarray,
) -> float:
    numerator = np.linalg.norm(analytical - numerical)
    denominator = max(
        1.0,
        np.linalg.norm(analytical),
        np.linalg.norm(numerical),
    )

    return float(numerator / denominator)


def main() -> None:
    point = np.array([2.3, -1.7])

    exact = analytical_gradient(point)
    estimated = numerical_gradient(loss, point)
    error = relative_error(exact, estimated)

    print(f"Analytical gradient: {exact}")
    print(f"Numerical gradient:  {estimated}")
    print(f"Relative error:      {error:.12e}")

    if error > 1e-6:
        raise RuntimeError("Gradient check failed")

    print("Gradient check passed")


if __name__ == "__main__":
    main()

Run it:

python gradient_check.py

A small relative error gives evidence that the implementation matches the derivative of the coded loss.

It does not prove that the loss itself represents your intended objective. If both your model specification and your analytical reasoning contain the same conceptual mistake, gradient checking will not detect it.

The cherry on the cake: a spiral valley

A bowl-shaped loss surface is useful for learning, but it is unusually friendly. The gradient always points toward the center in a predictable way.

Now consider a deliberately constructed surface with a spiral-shaped trough.

The optimizer has two jobs:

Move across the steep walls into the trough.
Follow the trough as it curves around the center.

Those goals can conflict.

A large step may quickly reach the valley but repeatedly overshoot its narrow floor. The gradient then points sharply back across the valley rather than gently along it. From the optimizer’s local perspective, crossing the valley wall is the most urgent correction, even though the long-term route bends around the spiral.

This is the surprising part: the direction of steepest immediate improvement does not have to point along the best long-distance route.

The following example is synthetic rather than a standard benchmark. Its purpose is to expose optimizer behavior that a simple quadratic bowl hides.

Create spiral_valley.py:

from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.animation import FuncAnimation, PillowWriter


def spiral_loss(
    points: np.ndarray,
    turn_rate: float = 3.0,
) -> np.ndarray:
    points = np.asarray(points, dtype=float)
    x = points[..., 0]
    y = points[..., 1]

    epsilon = 1e-6
    radius = np.sqrt(x * x + y * y + epsilon)
    angle = np.arctan2(y, x)
    phase = angle + turn_rate * radius

    spiral_penalty = 1.0 - np.cos(phase)
    inward_slope = 0.03 * radius * radius

    return spiral_penalty + inward_slope


def spiral_gradient(
    point: np.ndarray,
    turn_rate: float = 3.0,
) -> np.ndarray:
    x, y = np.asarray(point, dtype=float)

    epsilon = 1e-6
    radius_squared = x * x + y * y + epsilon
    radius = np.sqrt(radius_squared)
    angle = np.arctan2(y, x)
    phase = angle + turn_rate * radius

    phase_dx = (
        -y / radius_squared
        + turn_rate * x / radius
    )
    phase_dy = (
        x / radius_squared
        + turn_rate * y / radius
    )

    wave_derivative = np.sin(phase)

    return np.array(
        [
            wave_derivative * phase_dx + 0.06 * x,
            wave_derivative * phase_dy + 0.06 * y,
        ]
    )


def run_gradient_descent(
    start: tuple[float, float],
    learning_rate: float,
    steps: int,
) -> np.ndarray:
    path = [np.asarray(start, dtype=float)]

    for _ in range(steps):
        current = path[-1]
        next_point = (
            current
            - learning_rate * spiral_gradient(current)
        )
        path.append(next_point)

    return np.vstack(path)


def make_static_plot(
    x_grid: np.ndarray,
    y_grid: np.ndarray,
    loss_grid: np.ndarray,
    stable_path: np.ndarray,
    aggressive_path: np.ndarray,
) -> None:
    figure, axis = plt.subplots(figsize=(9, 8))

    axis.contour(
        x_grid,
        y_grid,
        loss_grid,
        levels=np.linspace(0.05, 2.1, 30),
    )

    axis.plot(
        stable_path[:, 0],
        stable_path[:, 1],
        linewidth=2,
        label="Learning rate 0.08",
    )

    axis.plot(
        aggressive_path[:, 0],
        aggressive_path[:, 1],
        linewidth=1.5,
        label="Learning rate 0.28",
    )

    axis.scatter(
        stable_path[0, 0],
        stable_path[0, 1],
        marker="s",
        s=90,
        label="Start",
    )

    axis.set_title("Two learning rates in a spiral-shaped valley")
    axis.set_xlabel("Parameter x")
    axis.set_ylabel("Parameter y")
    axis.set_aspect("equal", adjustable="box")
    axis.legend()

    output_path = Path("spiral_valley_paths.png")
    figure.tight_layout()
    figure.savefig(output_path, dpi=160)
    plt.show()

    print(f"Saved {output_path.resolve()}")


def make_animation(
    x_grid: np.ndarray,
    y_grid: np.ndarray,
    loss_grid: np.ndarray,
    path: np.ndarray,
) -> None:
    figure, axis = plt.subplots(figsize=(8, 8))

    axis.contour(
        x_grid,
        y_grid,
        loss_grid,
        levels=np.linspace(0.05, 2.1, 30),
    )

    trajectory, = axis.plot(
        [],
        [],
        linewidth=2,
        label="Optimizer path",
    )

    current_point, = axis.plot(
        [],
        [],
        marker="o",
        markersize=7,
    )

    status_text = axis.text(
        0.02,
        0.98,
        "",
        transform=axis.transAxes,
        verticalalignment="top",
    )

    axis.set_xlim(x_grid.min(), x_grid.max())
    axis.set_ylim(y_grid.min(), y_grid.max())
    axis.set_xlabel("Parameter x")
    axis.set_ylabel("Parameter y")
    axis.set_title("Gradient descent in a spiral-shaped valley")
    axis.set_aspect("equal", adjustable="box")
    axis.legend(loc="lower left")

    def initialize():
        trajectory.set_data([], [])
        current_point.set_data([], [])
        status_text.set_text("")
        return trajectory, current_point, status_text

    def update(frame: int):
        visible_path = path[: frame + 1]
        point = path[frame]

        trajectory.set_data(
            visible_path[:, 0],
            visible_path[:, 1],
        )

        current_point.set_data(
            [point[0]],
            [point[1]],
        )

        current_loss = float(spiral_loss(point))
        status_text.set_text(
            f"Step: {frame}\nLoss: {current_loss:.4f}"
        )

        return trajectory, current_point, status_text

    animation = FuncAnimation(
        figure,
        update,
        frames=len(path),
        init_func=initialize,
        interval=45,
        blit=True,
    )

    output_path = Path("spiral_descent.gif")
    writer = PillowWriter(fps=24)
    animation.save(output_path, writer=writer, dpi=110)
    plt.close(figure)

    print(f"Saved {output_path.resolve()}")


def count_uphill_steps(path: np.ndarray) -> int:
    losses = spiral_loss(path)
    return int(np.count_nonzero(np.diff(losses) > 0.0))


def main() -> None:
    x_values = np.linspace(-2.8, 2.8, 320)
    y_values = np.linspace(-2.8, 2.8, 320)
    x_grid, y_grid = np.meshgrid(x_values, y_values)

    grid_points = np.stack([x_grid, y_grid], axis=-1)
    loss_grid = spiral_loss(grid_points)

    start = (2.0, 1.5)

    stable_path = run_gradient_descent(
        start=start,
        learning_rate=0.08,
        steps=180,
    )

    aggressive_path = run_gradient_descent(
        start=start,
        learning_rate=0.28,
        steps=180,
    )

    print(
        "Stable final loss:",
        float(spiral_loss(stable_path[-1])),
    )
    print(
        "Aggressive final loss:",
        float(spiral_loss(aggressive_path[-1])),
    )
    print(
        "Stable uphill steps:",
        count_uphill_steps(stable_path),
    )
    print(
        "Aggressive uphill steps:",
        count_uphill_steps(aggressive_path),
    )

    make_static_plot(
        x_grid,
        y_grid,
        loss_grid,
        stable_path,
        aggressive_path,
    )

    make_animation(
        x_grid,
        y_grid,
        loss_grid,
        stable_path,
    )


if __name__ == "__main__":
    main()

Run it:

python spiral_valley.py

The script creates:

spiral_valley_paths.png
spiral_descent.gif

The stable path uses a learning rate of 0.08. The aggressive path uses 0.28.

On this surface, the aggressive path frequently takes steps that increase the loss. It bounces across the curved trough because its update is too large for the local width and curvature of the valley.

The animation is built with Matplotlib’s current FuncAnimation interface and saved through PillowWriter, which supports GIF output without requiring an external video encoder.

What the spiral teaches

The spiral is not difficult because the gradient is wrong. The gradient is locally correct at every point.

The difficulty comes from geometry:

The valley is narrow.
The useful route is curved.
Curvature changes the gradient direction from step to step.
A fixed learning rate ignores those geometric changes.
The steepest local direction may point mostly across the valley.
Long-term progress requires movement along the valley.

This helps explain why optimizer design matters even when derivatives are exact.

What more advanced optimizers change

Basic gradient descent uses only the current gradient:

update = -learning rate × current gradient

More advanced methods alter the update without changing the underlying derivative.

Momentum

Momentum maintains a running velocity. Consistent gradient directions build speed, while alternating directions partially cancel.

In a narrow valley, the gradient may alternate strongly across the walls but remain more consistent along the valley. Momentum can reduce some of the side-to-side zig-zag while preserving forward movement.

Adaptive per-parameter scaling

Adaptive optimizers maintain statistics about recent gradients and scale updates differently for different parameters.

This can help when:

Features use different numerical scales.
Some parameters receive consistently large gradients.
Some parameters receive sparse gradients.
The loss surface has different curvature along different coordinates.

These methods do not eliminate the need for a suitable learning rate. They change how that learning rate is distributed across parameters and time.

Learning-rate schedules

A schedule changes the learning rate during training.

A common strategy is:

Use larger updates early, when the parameters are far from a useful solution.
Reduce the learning rate later, when large updates would bounce around a narrow minimum.

The important idea is not a particular schedule. It is that the best update scale may change as optimization moves through different parts of the loss surface.

Derivative, gradient, and optimization are different things

These terms are related but not interchangeable.

Derivative

A derivative measures local change with respect to one variable:

dL/dw

Partial derivative

A partial derivative measures local change with respect to one variable in a multivariable function:

∂L/∂w₁

Gradient

A gradient collects all relevant partial derivatives:

∇L = [∂L/∂w₁, ∂L/∂w₂, ..., ∂L/∂wₙ]

Gradient descent

Gradient descent is an algorithm that uses the gradient:

parameters = parameters - learning rate × gradient

Calculating a gradient does not train a model by itself. Training requires repeatedly evaluating the model, computing the loss, obtaining gradients, and applying updates.

A practical debugging checklist

When a gradient-based model will not train, inspect the problem systematically.

Check the loss

Is the loss finite?
Does it have the expected scale?
Does a better prediction produce a lower loss?
Are reduction operations averaging over the intended dimensions?

Check the data

Are feature scales wildly different?
Are input values finite?
Are targets aligned with the correct examples?
Are array shapes what you expect?
Has broadcasting silently produced an unintended tensor?

Check the gradient

Is its shape identical to the parameter shape?
Are its values finite?
Is it exactly zero everywhere?
Is its magnitude implausibly large?
Does a finite-difference gradient check pass on a small example?

Check the update

Is the gradient being subtracted rather than added?
Is the learning rate nonzero?
Are parameters actually being modified?
Are gradients accidentally reused from an earlier step?
Is the optimizer updating the intended parameter collection?

Check the experiment

Can the model overfit a tiny dataset?
Does reducing the learning rate stop divergence?
Does increasing the learning rate improve an extremely slow run?
Does standardizing the features improve the trajectory?
Does plotting the loss reveal oscillation or instability?

A tiny overfitting test is especially powerful. A sufficiently expressive model should usually be able to drive the training loss very low on a handful of examples. Failure often indicates an implementation, data, or optimization problem rather than a lack of model capacity.

The mental model to keep

You do not need every technique from a full calculus course to understand the core of machine-learning optimization.

Keep this sequence in mind:

The model transforms inputs into predictions.
The loss scores those predictions.
A derivative measures how one parameter affects the loss.
The chain rule carries that sensitivity through nested operations.
A gradient collects the sensitivities for all parameters.
The optimizer converts the gradient into an update.
Repeating the process gradually changes the model’s behavior.

The central update remains:

new parameters = old parameters - learning rate × gradient

Everything else—automatic differentiation, backpropagation, momentum, adaptive scaling, schedules, normalization, clipping, and second-order methods—builds around the challenges of computing or using that update effectively.

Put the lesson into practice

Run all five programs, then modify them rather than merely reading them:

Change the starting points.
Try learning rates above and below the provided values.
Add noise to the synthetic regression data.
Deliberately break an analytical gradient and watch the checker fail.
Make the quadratic surface narrower by increasing the coefficient on one parameter.
Change the spiral’s turn_rate and observe how curvature alters the optimizer’s path.
Add momentum to the spiral example and compare its trajectory with plain gradient descent.

The fastest way to make derivatives and gradients feel intuitive is to watch parameter updates move across a surface you can see. Start with the contour scripts, create one modification of your own, and explain—in plain language—why the resulting path changed.

What you will learn

Set up the runnable environment

A derivative answers a local question

The derivative is not the function value

Estimating a derivative numerically

Why not make h infinitely small?

From derivatives to partial derivatives

A gradient packages the partial derivatives

Gradient descent in one dimension

The chain rule is the part machine learning really needs

Training a linear model from scratch

Where the batch gradient comes from

Why standardize the input?

Visualizing descent on a two-dimensional loss surface

What the learning rate really controls

When the learning rate is too small

When the learning rate is too large

When the learning rate is reasonable

Gradient checking catches implementation mistakes

The cherry on the cake: a spiral valley

What the spiral teaches

What more advanced optimizers change

Momentum

Adaptive per-parameter scaling

Learning-rate schedules

Derivative, gradient, and optimization are different things

Derivative

Partial derivative

Gradient

Gradient descent

A practical debugging checklist

Check the loss

Check the data

Check the gradient

Check the update

Check the experiment

The mental model to keep

Put the lesson into practice

Random forests: why averaging many bad trees makes one great model

Vulnerability Watch No33

Correlation is not causation, but it’s also not nothing: reading correlation matrices right

Why not make `h` infinitely small?