How diffusion models generate images, explained with a runnable toy

LEARN · COMPUTER VISION

Diffusion models are among the most important ideas in modern GenerativeAI and ComputerVision. They power many image-generation systems by learning a simple but powerful concept:

Start with structured data, such as an image.
Gradually destroy the structure by adding noise.
Train a neural network to reverse that corruption.
Generate new content by repeatedly removing predicted noise.

The result is not an image lookup system. A diffusion model learns statistical patterns about visual data and uses those patterns to transform random noise into a new, plausible sample.

This post explains the denoising process, builds a tiny runnable diffusion-style experiment, and shows why sampling strategy matters.

The core idea: learning to remove noise

Imagine taking a photo and adding increasing amounts of static.

At low noise levels:

Shapes remain recognizable.
Colors are still visible.
Fine details begin to disappear.

At high noise levels:

Objects become unclear.
The original image is difficult to recover.
The result approaches random noise.

Diffusion training creates these noisy examples intentionally. The model learns the reverse process:

Take a noisy input.
Predict the noise that was added.
Remove part of that noise.
Repeat until a cleaner signal appears.

During generation, the model does not receive the original image. It begins with random noise and uses learned visual patterns to guide the denoising process.

Forward diffusion: adding controlled noise

A common diffusion formulation creates a noisy version of an image with:

xₜ = √αₜ · x₀ + √(1 – αₜ) · ε

Where:

x₀ is the original clean image.
xₜ is the image after adding noise at timestep t.
αₜ controls how much original information remains.
ε is random noise.

The noise schedule determines how quickly information disappears.

A useful schedule creates many intermediate examples:

Slightly corrupted images.
Medium-noise images.
Almost pure noise.

These examples teach the model how images transition between order and randomness.

What the neural network learns

Most diffusion models are trained to predict the noise component rather than directly predict the final clean image.

The network receives:

A noisy image representation.
A timestep describing the current noise level.
Optional conditioning information, such as text embeddings.

It outputs:

A prediction of the noise added during corruption.

Training compares the predicted noise with the actual noise.

A simplified training step can be written as a complete PyTorch example:

import torch
import torch.nn as nn

def add_noise(clean_images, noise, timesteps, alpha_bars):
    selected_alpha = alpha_bars[timesteps].reshape(-1, 1)
    noisy_images = (
        torch.sqrt(selected_alpha) * clean_images
        + torch.sqrt(1 - selected_alpha) * noise
    )
    return noisy_images

model = nn.Sequential(
    nn.Linear(1, 64),
    nn.ReLU(),
    nn.Linear(64, 1)
)

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

clean_images = torch.randn(32, 1)

alpha_bars = torch.linspace(1.0, 0.1, 100)

timesteps = torch.randint(
    0,
    len(alpha_bars),
    (32,)
)

noise = torch.randn_like(clean_images)

noisy_images = add_noise(
    clean_images,
    noise,
    timesteps,
    alpha_bars
)

predicted_noise = model(noisy_images)

loss = torch.mean(
    (predicted_noise - noise) ** 2
)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print("Training loss:", float(loss))

Real image-generation systems add many more components:

Large image datasets.
Much larger neural networks.
Attention mechanisms.
Text encoders.
Specialized samplers.
GPU acceleration.

The underlying learning principle remains the same: predict how to move from noise toward structure.

A tiny runnable diffusion-style experiment

A complete text-to-image model requires significant compute, but the core idea can be demonstrated with a small one-dimensional example.

This program:

Creates simple clean data.
Adds random noise.
Trains a network to predict that noise.
Starts from random noise and applies denoising steps.

Install PyTorch if needed:

pip install torch

Run the experiment:

import torch
import torch.nn as nn
import torch.optim as optim

torch.manual_seed(0)

clean = torch.linspace(-1, 1, 200).reshape(-1, 1)

model = nn.Sequential(
    nn.Linear(1, 32),
    nn.ReLU(),
    nn.Linear(32, 1)
)

optimizer = optim.Adam(
    model.parameters(),
    lr=0.01
)

for step in range(1000):
    noise = torch.randn_like(clean) * 0.3
    noisy = clean + noise

    predicted_noise = model(noisy)

    loss = torch.mean(
        (predicted_noise - noise) ** 2
    )

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print("Final training loss:", float(loss))

sample = torch.randn(1, 1)

for _ in range(20):
    predicted_noise = model(sample)
    sample = sample - predicted_noise * 0.1

print("Generated sample:", float(sample))

This is not an image generator, but it demonstrates the essential mechanism:

Data has a learned distribution.
Noise pushes samples away from that distribution.
The model learns a direction back toward meaningful structure.

From text prompts to generated images

Modern image diffusion systems add conditioning.

Instead of learning only:

How do I remove noise?

The model learns:

How do I remove noise while matching this description?

A simplified pipeline looks like this:

A text encoder converts a prompt into numerical representations.
A diffusion model processes noisy image information.
Attention layers connect text features with visual features.
The model gradually denoises the output.

A prompt such as:

A red bicycle beside a snowy mountain lake

does not point to a stored photograph. The model has learned relationships between:

Words.
Shapes.
Textures.
Colors.
Visual compositions.

Many practical systems use latent diffusion. Instead of operating directly on every pixel, the model works in a compressed representation of the image. This reduces computational requirements while preserving important visual information.

Why sampling steps matter

Generation requires multiple denoising steps. Each step usually requires another neural network prediction, so the number of steps directly affects speed.

A simplified sampler looks like this:

import torch

def generate(model, steps=50):
    image = torch.randn(
        1,
        3,
        256,
        256
    )

    for step in reversed(range(steps)):
        predicted_noise = model(
            image,
            step
        )

        image = image - predicted_noise * 0.02

    return image

More steps can provide:

More opportunities for refinement.
Better results with some model and scheduler combinations.
Improved stability in certain workflows.

Fewer steps can provide:

Faster generation.
Lower compute costs.
Better interactive experiences.

However, more steps do not automatically produce better images.

Quality depends on:

The trained model.
The sampling algorithm.
The noise schedule.
The requested resolution.
The specific generation task.

Cherry on the cake: why 1000 steps are not always better

Early diffusion systems often needed hundreds or thousands of denoising steps, which made generation expensive.

A surprising shift came from research into improved sampling methods. The Denoising Diffusion Implicit Models approach showed that diffusion models could generate high-quality samples with fewer carefully designed steps by changing the sampling process.

The lesson is important:

A 1000-step generation is not automatically better than a 50-step generation.

Modern diffusion workflows focus on better steps rather than simply more steps. A well-designed sampler can produce strong results faster than a naive long sampling process.

Diffusion models are not image search engines

A common misunderstanding is that image generators simply retrieve training images.

Instead, training adjusts model parameters so the network learns statistical relationships from examples. The model stores patterns in its parameters rather than maintaining a traditional database of images to copy.

Training data still has major effects:

Higher-quality data can improve results.
Dataset biases can influence outputs.
Data selection affects safety and fairness discussions.

The technology is shaped by both engineering decisions and the data used to train it.

Where diffusion models are going

Recent progress has focused on making diffusion systems:

Faster.
Smaller.
Easier to control.
Better at editing.
More capable across different media types.

Current research and products increasingly explore:

Efficient sampling methods.
Models that run on local hardware.
Better image consistency.
Combined text, image, audio, and video generation.

The main idea remains unchanged:

Learn how to transform randomness into meaningful structure.

Key takeaways

Diffusion models learn to reverse a controlled noise process.
Training usually teaches a network to predict and remove noise.
Image generation begins with random noise, not a stored picture.
Text conditioning guides the denoising process toward specific concepts.
Sampling quality matters more than simply increasing step counts.
A small experiment can reveal the same principles behind large-scale systems.

Your next experiment

Take the toy program and modify it.

Try:

Adding timestep information to the model.
Changing the noise level.
Visualizing intermediate noisy states.
Comparing different denoising step counts.
Measuring how sampling changes the final result.

Run the experiments, inspect the outputs, and build your understanding of generative models by turning diffusion theory into working code.