Diffusion,
from noise to image.

A generative model has one job: produce new samples that look like they came from the training data. Diffusion models approach this sideways. Instead of learning to build an image from nothing, they first define a process that destroys images in a very controlled way, and then learn to run that process in reverse.

The destruction is a Markov chain. Take an image $x_0$ with pixels scaled to $[-1,1]$. Shrink it slightly and add a small amount of Gaussian noise to get $x_1$. Do the same to $x_1$ to get $x_2$. Keep going for $T$ steps:

${\beta }_t$ is a small variance, typically growing linearly from $10^{-4}$ to $0.02$ over $T=1000$ steps. The $\sqrt{1-{\beta }_t}$ factor is not decoration. If $x_{t-1}$ has unit variance, then $x_t$ has variance $(1-{\beta }_t)+{\beta }_t=1$. The chain shuffles variance from signal to noise without ever changing the total, which is why it is called variance preserving, and why after enough steps $x_T$ is simply a standard Gaussian, with no trace of the image left in it.

import torch

T = 1000
betas = torch.linspace(1e-4, 0.02, T)     # beta_1 ... beta_T, tiny variances

def forward_step(x_prev, t):
    """One step of the Markov chain: q(x_t | x_{t-1})."""
    beta = betas[t - 1]
    eps = torch.randn_like(x_prev)          # fresh Gaussian noise every step
    return (1 - beta).sqrt() * x_prev + beta.sqrt() * eps

def diffuse(x0, steps):
    """Walk the chain from a clean image x0 to x_steps."""
    x = x0
    trajectory = [x0]
    for t in range(1, steps + 1):
        x = forward_step(x, t)
        trajectory.append(x)
    return trajectory                       # x_0, x_1, ..., x_steps

# x0: an image scaled to [-1, 1], shape (3, H, W)
# diffuse(x0, 1000)[-1] is indistinguishable from torch.randn_like(x0)

Two things are worth noticing in the demo. The image survives far longer than the per-step noise would suggest, because each ${\beta }_t$ is tiny. And the endpoint does not depend on the input at all: swap in your own picture and the histogram at $t=1000$ is the same bell curve. That second point is the whole trick. If every image ends in the same place, then a model that can walk the chain backwards can start from that place, pure noise, and arrive at an image.

The forward process has no parameters and nothing to learn. It is a fixed recipe. Everything a diffusion model learns is about the reverse direction, and to train that efficiently we first need a shortcut through the chain.

Simulating a thousand steps to get one noisy image would make training hopeless. Fortunately the chain folds. A Gaussian scaled by a constant is Gaussian, and the sum of two independent Gaussians is Gaussian with the variances added. So the composition of $t$ noising steps is itself a single Gaussian step, and its parameters have a clean form. Write ${\alpha }_t=1-{\beta }_t$ and ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$. Then:

which is the same as saying

You can check the two-step case by hand. After step one, $x_1=\sqrt{{\alpha }_1}{x}_0+\sqrt{{\beta }_1}{ϵ}_1$. After step two, $x_2=\sqrt{{\alpha }_2{\alpha }_1}{x}_0+\sqrt{{\alpha }_2{\beta }_1}{ϵ}_1+\sqrt{{\beta }_2}{ϵ}_2$. The two noise terms are independent, so their variances add: ${\alpha }_2{\beta }_1+{\beta }_2={\alpha }_2(1-{\alpha }_1)+(1-{\alpha }_2)=1-{\alpha }_1{\alpha }_2$. That is $1-{\bar{\alpha }}_2$, and induction does the rest.

This single equation is the reason diffusion is practical. During training you pick a random $t$, draw one $ϵ$, and produce $x_t$ in one line. No chain, no loop. ${\bar{\alpha }}_t$ is simply “how much of the original signal is left after $t$ steps”, and every schedule, sampler, and loss on this page is written in terms of it.

import torch

def linear_beta_schedule(T=1000, beta_start=1e-4, beta_end=2e-2):
    return torch.linspace(beta_start, beta_end, T)


def precompute_schedule(betas):
    alphas = 1.0 - betas
    alphas_cumprod = torch.cumprod(alphas, dim=0)        # alpha_bar_t
    return {
        "betas": betas,
        "alphas": alphas,
        "alphas_cumprod": alphas_cumprod,
        "sqrt_alphas_cumprod": torch.sqrt(alphas_cumprod),
        "sqrt_one_minus_alphas_cumprod": torch.sqrt(1.0 - alphas_cumprod),
        "sqrt_recip_alphas": torch.sqrt(1.0 / alphas),
    }


def q_sample(x0, t, noise, schedule):
    """Jump straight from x0 to x_t. t is a batch of timesteps."""
    sqrt_a = schedule["sqrt_alphas_cumprod"][t].view(-1, 1, 1, 1)
    sqrt_one_minus_a = schedule["sqrt_one_minus_alphas_cumprod"][t].view(-1, 1, 1, 1)
    return sqrt_a * x0 + sqrt_one_minus_a * noise


schedule = precompute_schedule(linear_beta_schedule(T=1000))
# alpha_bar[  0] = 0.9999   alpha_bar[499] = 0.0655   alpha_bar[999] = 0.00004

Another way to read the closed form: a diffusion model never sees the chain during training. It only ever sees $(x_t,t)$ pairs produced by the jump, drawn at random timesteps. The chain matters for the theory, and for sampling, but the training set is just “clean image, mixed with noise, in a known ratio”.

The forward process is defined entirely by the sequence ${\beta }_1,\dots ,{\beta }_T$, or equivalently by the curve ${\bar{\alpha }}_t$. That curve decides how much signal is left at every timestep, and therefore what the network is asked to do at every timestep. DDPM used a linear $\beta$ ramp, and it worked, but if you plot ${\bar{\alpha }}_t$ for it you find a problem: the signal is essentially gone by $t\approx 600$. The last third of the chain is spent turning noise into slightly different noise.

Every training step samples $t$ uniformly, so a third of the training budget goes to timesteps where there is nothing to learn. Sampling has the same waste in reverse. Nichol and Dhariwal proposed defining ${\bar{\alpha }}_t$ directly as a cosine curve and deriving the ${\beta }_t$ from it, which spreads the destruction more evenly across the chain. Sigmoid schedules do something similar with a tunable steepness. A useful single number for comparing them is the signal-to-noise ratio $\mathrm{SNR}(t)={\bar{\alpha }}_t/(1-{\bar{\alpha }}_t)$, usually plotted in decibels.

import math
import torch

def linear_betas(T=1000, beta_start=1e-4, beta_end=0.02):
    return torch.linspace(beta_start, beta_end, T)


def cosine_betas(T=1000, s=0.008):
    """Nichol & Dhariwal (2021). Define alpha_bar directly, derive betas."""
    steps = torch.arange(T + 1, dtype=torch.float64)
    f = torch.cos(((steps / T + s) / (1 + s)) * math.pi / 2) ** 2
    alpha_bar = f / f[0]                               # alpha_bar_0 = 1
    betas = 1 - alpha_bar[1:] / alpha_bar[:-1]
    return betas.clamp(max=0.999).float()


def sigmoid_betas(T=1000, k=8.0):
    steps = torch.arange(T + 1, dtype=torch.float64)
    g = torch.sigmoid(k * (0.5 - steps / T))
    alpha_bar = (g - g[-1]) / (g[0] - g[-1])          # runs from 1 to 0
    betas = 1 - alpha_bar[1:] / alpha_bar[:-1]
    return betas.clamp(1e-8, 0.999).float()


def snr(betas):
    alpha_bar = torch.cumprod(1 - betas, dim=0)
    return alpha_bar / (1 - alpha_bar)

# where does the signal effectively vanish (alpha_bar < 0.01)?
for name, b in [("linear", linear_betas()), ("cosine", cosine_betas()), ("sigmoid", sigmoid_betas())]:
    ab = torch.cumprod(1 - b, dim=0)
    print(name, int((ab > 0.01).sum()), "of 1000 steps still carry signal")
# linear   ~ 620   <- the last ~38% of the chain is already pure noise
# cosine   ~ 936
# sigmoid  ~ 790

Two further details matter in practice. First, the schedule should actually reach zero signal. Stable Diffusion 1.x used a “scaled linear” schedule whose $\sqrt{{\bar{\alpha }}_T}\approx 0.068$, meaning the model never saw pure noise during training but was fed exactly that at inference. The mismatch shows up as an inability to produce very dark or very bright images, and the fix (Lin et al., 2023) is to rescale the schedule to zero terminal SNR. Second, the right schedule depends on resolution. Adding the same per-pixel noise to a 1024-pixel image destroys less information than adding it to a 64-pixel one, because neighbouring pixels are redundant, so high-resolution models shift their schedules toward more noise. Rectified-flow models in section 11 take this to its conclusion and make the schedule a straight line.

Running the chain backwards means sampling from $p(x_{t-1}\mid x_t)$, and that distribution is not known in general. Two facts rescue us. When ${\beta }_t$ is small, the reverse conditional is close to Gaussian. And if you also know the clean image $x_0$, the reverse conditional $q(x_{t-1}\mid x_t,x_0)$ is exactly Gaussian, with a mean that Bayes’ rule gives in closed form. So a network that can estimate $x_0$ from $x_t$ can estimate the mean of the reverse step.

DDPM does not ask the network for $x_0$ directly. It asks for the noise $ϵ$ that was mixed in, since the closed form $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$ makes the two interchangeable:

Why noise and not the image? Partly empirical: Ho et al. found it trained better. Partly structural: at small $t$ the noise is a small perturbation and predicting it is a well-scaled regression, whereas predicting $x_0$ would mean reproducing the input almost exactly. The catch is at large $t$, where $\sqrt{{\bar{\alpha }}_t}$ is tiny and the division above amplifies any error in ${ϵ}_{\theta }$ enormously. That is why samplers take many small steps and re-predict every time rather than trusting one ${\hat{x}}_0$, and why later work introduced $v$-prediction, a rotation of the $(x_0,ϵ)$ pair that stays well-conditioned at both ends.

import torch

def to_x0(x_t, eps, alpha_bar):
    """Every parameterization is a change of variables around this identity."""
    return (x_t - (1 - alpha_bar).sqrt() * eps) / alpha_bar.sqrt()


def to_eps(x_t, x0, alpha_bar):
    return (x_t - alpha_bar.sqrt() * x0) / (1 - alpha_bar).sqrt()


def to_v(x0, eps, alpha_bar):
    """v-prediction (Salimans & Ho, 2022): a rotation of (x0, eps)."""
    return alpha_bar.sqrt() * eps - (1 - alpha_bar).sqrt() * x0


def v_to_x0_eps(x_t, v, alpha_bar):
    a, b = alpha_bar.sqrt(), (1 - alpha_bar).sqrt()
    x0  = a * x_t - b * v
    eps = b * x_t + a * v
    return x0, eps


def posterior_mean(x_t, eps, t, sch):
    """mu_theta(x_t, t): the mean of p(x_{t-1} | x_t) given a noise estimate."""
    beta = sch["betas"][t]
    coef = beta / sch["sqrt_one_minus_alphas_cumprod"][t]
    return sch["sqrt_recip_alphas"][t] * (x_t - coef * eps)

# Score-function view: the noise estimate is a scaled gradient of log density.
#   eps_theta(x_t, t) = -sqrt(1 - alpha_bar_t) * grad_x log p_t(x_t)
# Predicting noise and predicting "which way is more probable" are the same job.

One more reading of the same object, because it shows up constantly in the literature. Since $q(x_t\mid x_0)$ is Gaussian, the gradient of its log density with respect to $x_t$ is $-ϵ/\sqrt{1-{\bar{\alpha }}_t}$. Averaging over the data, the optimal noise predictor is a scaled version of ${\nabla }_x\log p_t(x)$, the score of the noisy data distribution. Predicting the noise, predicting the clean image, and predicting which direction increases the probability of $x_t$ are three coordinate systems for one function. Song et al. built the same models from the score side and showed DDPM is a discretization of a stochastic differential equation, which is what makes the fast ODE samplers in section 08 possible.

Whatever the parameterization, the model is a function ${ϵ}_{\theta }(x_t,t)$ that maps an image-shaped tensor to an image-shaped tensor. Any architecture that preserves spatial shape works. Convolutional U-Nets were the choice from 2015 through the Stable Diffusion era, and transformers on patches (DiT) have taken over since; section 11 covers that shift. The U-Net is still the clearest way to see what the denoiser needs.

It needs three things. First, a way to look at large regions: the encoder halves the resolution at each level so deep layers see the whole image and can decide that a blob of pixels is a petal. Second, a way to keep fine detail: skip connections copy each encoder level straight across to the matching decoder level, so the output can be sharp even though the bottleneck is coarse. Third, the timestep. The same network must remove a whisper of noise at $t=20$ and hallucinate global structure from static at $t=900$. Without knowing $t$ it would have to infer the noise level from the input, which is possible but wasteful.

The timestep enters through a sinusoidal embedding, the same trick transformers use for positions. Each of the embedding’s $d$ channels is a sine or cosine of $t$ at a different frequency, from one full cycle per step to one per ten thousand steps. Nearby timesteps get nearby vectors, every timestep gets a distinct one, and a small MLP turns the vector into a per-channel shift (or a scale and shift, called FiLM or adaptive normalization) added to the feature maps at every level.

import math
import torch
import torch.nn as nn
import torch.nn.functional as F

def timestep_embedding(t, dim=64):
    """Sinusoidal embedding of an integer timestep, transformer-style."""
    half = dim // 2
    freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device) / half)
    args = t[:, None].float() * freqs[None]
    return torch.cat([args.sin(), args.cos()], dim=-1)


class TinyUNet(nn.Module):
    def __init__(self, img_channels=3, base=32, t_dim=64):
        super().__init__()
        self.t_dim = t_dim
        self.t_mlp = nn.Sequential(nn.Linear(t_dim, base * 4), nn.SiLU(), nn.Linear(base * 4, base * 4))
        self.time_proj = nn.Linear(base * 4, base * 2)

        self.enc1 = nn.Conv2d(img_channels, base, 3, padding=1)           # H x W
        self.enc2 = nn.Conv2d(base, base * 2, 4, stride=2, padding=1)      # H/2 x W/2
        self.mid  = nn.Conv2d(base * 2, base * 2, 3, padding=1)
        self.dec1 = nn.ConvTranspose2d(base * 2, base, 4, stride=2, padding=1)
        self.dec2 = nn.Conv2d(base * 2, img_channels, 3, padding=1)        # cat(skip, up) -> eps

    def forward(self, x, t):
        t_emb = self.t_mlp(timestep_embedding(t, self.t_dim))
        t_proj = self.time_proj(t_emb)[:, :, None, None]   # broadcast over H, W

        h1 = F.silu(self.enc1(x))
        h2 = F.silu(self.enc2(h1)) + t_proj                # time enters here
        h3 = F.silu(self.mid(h2))
        d1 = F.silu(self.dec1(h3))
        d2 = torch.cat([d1, h1], dim=1)                    # skip connection
        return self.dec2(d2)                               # same shape as x

Real diffusion U-Nets add self-attention blocks at the lower resolutions so distant pixels can coordinate, and (for text conditioning) cross-attention blocks that let each spatial position read from the prompt’s token embeddings. Stable Diffusion 1.5’s U-Net has about 860M parameters, most of them in those attention layers. The convolutions are the cheap part.

Put the pieces together and the training loop is almost embarrassingly short. Take a batch of clean images. For each one draw a random timestep $t$ and a random $ϵ$. Use the closed form to build $x_t$. Ask the network for the noise. Take the mean squared error between its answer and the $ϵ$ you actually used:

Ho et al. derived this by starting from the variational bound that Sohl-Dickstein’s 2015 paper had used, working through the KL terms between the true reverse conditionals and the model’s, and noticing that with the noise parameterization every term becomes a weighted squared error on $ϵ$. Then they dropped the weights. The unweighted version trains better, and it is equivalent to reweighting the objective toward the middle timesteps, where the interesting perceptual work happens.

Pixels are expensive to train on in a browser tab, so the demo below swaps them for 2D points. The math does not care: a point is a two-dimensional image, the noise schedule is the same linear one, and the network is a small MLP that takes $(x,y)$, the sinusoidal embedding of $t$, and a class label. Around ten thousand parameters, trained live with hand-written backprop. Watch the loss fall and the samples sharpen.

import torch
import torch.nn.functional as F

def train_step(model, x0, schedule, optimizer, device, T=1000):
    """Algorithm 1 of the DDPM paper. One batch, one optimizer step."""
    model.train()
    x0 = x0.to(device)
    bs = x0.size(0)

    t = torch.randint(0, T, (bs,), device=device)      # 1. random timestep per image
    noise = torch.randn_like(x0)                        # 2. random noise
    x_t = q_sample(x0, t, noise, schedule)              # 3. jump to x_t (closed form)

    pred = model(x_t, t)                                # 4. predict the noise
    loss = F.mse_loss(pred, noise)                      # 5. plain MSE

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    return loss.item()

# That is the entire training loop. No discriminator, no adversarial game,
# no mode collapse. The loss goes down and stays down.

Notice what the loss does. It drops quickly to around 0.3 and then creeps down slowly, and it never reaches zero. It cannot: at large $t$ the input is nearly pure noise and the best possible prediction of $ϵ$ is only slightly better than a guess. The irreducible part of the loss is the entropy of the data, not a failure of the model. This is also why a diffusion loss curve is a poor guide to sample quality, and why people look at samples (or FID) rather than the number.

With a trained noise predictor, generation is the reverse loop. Start from $x_T\sim \mathcal{N}(0,\mathbf{I})$. At each step ask the network for ${ϵ}_{\theta }(x_t,t)$, compute the posterior mean from section 04, add a fresh Gaussian kick of variance ${\sigma }_t^2={\beta }_t$, and move to $x_{t-1}$:

The added noise is not a bug. Each reverse step is a sample from a distribution, not a point estimate, and skipping the ${\sigma }_{t}z$ term (while keeping the DDPM update) produces blurry, over-averaged output. The randomness is also what makes two runs from the same starting noise diverge: DDPM is an ancestral sampler, every step branches.

The demo runs the actual loop on the toy model from the previous section, one thousand network calls per run. The solid points are $x_t$. The faint points are the model’s current guess of the clean data, ${\hat{x}}_0$, recomputed at every step. Early on that guess is a smear near the origin; as the noise level drops it snaps onto the shape while $x_t$ is still visibly noisy. That gap between “what the model believes” and “where the sample currently is” is exactly what the faster samplers in the next section exploit.

@torch.no_grad()
def sample_ddpm(model, schedule, shape, T=1000, device="cpu"):
    """Algorithm 2 of the DDPM paper: ancestral sampling, T network calls."""
    model.eval()
    x = torch.randn(shape, device=device)                     # x_T ~ N(0, I)
    betas = schedule["betas"].to(device)
    sqrt_one_minus_a = schedule["sqrt_one_minus_alphas_cumprod"].to(device)
    sqrt_recip_alphas = schedule["sqrt_recip_alphas"].to(device)

    for t in reversed(range(T)):
        t_batch = torch.full((shape[0],), t, dtype=torch.long, device=device)
        eps = model(x, t_batch)                               # one forward pass
        coef = betas[t] / sqrt_one_minus_a[t]
        mean = sqrt_recip_alphas[t] * (x - coef * eps)        # mu_theta(x_t, t)
        if t > 0:
            x = mean + torch.sqrt(betas[t]) * torch.randn_like(x)   # add sigma_t z
        else:
            x = mean                                          # last step: no noise
    return x

# 1000 forward passes per batch of samples. Correct, slow, and the
# baseline every faster sampler is measured against.

For images the same loop costs a thousand U-Net evaluations per sample, tens of seconds on a GPU. That is the reason the original DDPM paper was greeted with “impressive, but who would use this”, and the reason the next two years of research were mostly about doing it in fifty steps, then twenty, then four.

Song, Meng and Ermon noticed that the training objective only ever uses $q(x_t\mid x_0)$, the marginals of the forward process. It never uses the fact that the chain is Markov. So you are free to define a different, non-Markovian forward process with the same marginals, and its reverse process is a different sampler for the same trained network. One family of such processes has a knob $\eta$. At $\eta =1$ you get DDPM back. At $\eta =0$ the reverse step becomes deterministic.

The deterministic update reads naturally. Predict the noise, use it to estimate ${\hat{x}}_0$, then move directly to the point that ${\hat{x}}_0$ would occupy at the next noise level:

Because there is no injected noise, nothing forces $t^{\prime }$ to be $t-1$. You can jump from 1000 to 980 to 960 and the update is still consistent. Fifty steps give samples close to the thousand-step DDPM result. The same determinism means a fixed $x_T$ always produces the same image, which is what makes interpolation in noise space and image editing by inversion possible.

The deeper view is that the deterministic sampler is integrating an ordinary differential equation, the probability flow ODE that shares its marginals with the diffusion SDE. Once you see it as an ODE, the whole numerical-methods toolbox applies: Euler, Heun, multistep methods. DPM-Solver++ and the Karras samplers are higher-order solvers tuned to the specific shape of this ODE, and they reach DDIM-50 quality in twenty steps or fewer. Every production pipeline in 2026 uses one of them.

@torch.no_grad()
def sample_ddim(model, schedule, shape, steps=50, T=1000, device="cpu", eta=0.0):
    """Song et al. (2020). Same trained model, far fewer steps."""
    model.eval()
    x = torch.randn(shape, device=device)
    alphas_cumprod = schedule["alphas_cumprod"].to(device)

    ts = torch.linspace(T - 1, 0, steps + 1).long()      # a sparse subset of timesteps
    for i in range(steps):
        t, t_prev = int(ts[i]), int(ts[i + 1])
        t_batch = torch.full((shape[0],), t, dtype=torch.long, device=device)
        eps = model(x, t_batch)

        a_t, a_prev = alphas_cumprod[t], alphas_cumprod[t_prev]
        x0_pred = (x - torch.sqrt(1 - a_t) * eps) / torch.sqrt(a_t)     # "where is this going"
        sigma = eta * torch.sqrt((1 - a_prev) / (1 - a_t) * (1 - a_t / a_prev))
        dir_xt = torch.sqrt(1 - a_prev - sigma ** 2) * eps              # "re-noise to level t_prev"
        noise = sigma * torch.randn_like(x) if eta > 0 else 0
        x = torch.sqrt(a_prev) * x0_pred + dir_xt + noise
    return x

# eta = 0: deterministic. Same x_T always gives the same x_0.
# eta = 1: recovers DDPM's stochastic update on the chosen subsequence.

What DDIM cannot do is go much below ten steps on a model trained with the DDPM objective. The ODE trajectories curve, and a coarse Euler-style walk along a curved path drifts off it. Fixing that needs either a better solver, a straighter path (section 11), or a model distilled to take big steps on purpose.

Everything so far works on pixels, and pixels are expensive. A 512×512 RGB image is 786,432 numbers, every U-Net call touches all of them at full resolution, and the model spends most of its capacity on imperceptible high-frequency detail. Rombach et al. made the observation that turned diffusion into a product: most of those bits are perceptually irrelevant, so compress first, then diffuse in the compressed space.

The compressor is a variational autoencoder. The encoder maps a $3\times 512\times 512$ image to a $4\times 64\times 64$ latent, an 8× spatial reduction and 48× fewer numbers. The decoder maps it back. It is trained with a reconstruction loss, a perceptual loss, a small GAN loss for sharpness, and a very weak KL penalty that keeps the latents roughly Gaussian so the diffusion model has something well-behaved to noise. Once trained it is frozen. The diffusion model never sees a pixel again: it is trained on latents, samples latents, and the decoder is the last thing that runs.

The latent is not a small image. Decoding a random $4\times 64\times 64$ tensor gives garbage, because only a thin manifold of latents decodes to valid pictures. But it behaves like an image in the ways that matter: nearby latents decode to nearby images, and the closed-form noising still applies. That is what makes image-to-image possible. Encode a photo, add noise up to $t=\text{strength}\cdot T$, and denoise from there: the composition survives, the details are regenerated under the new prompt. Inpainting is the same idea with a mask deciding which latent positions get overwritten each step.

import torch
from diffusers import AutoencoderKL

vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse").to("cuda", torch.float16)
SCALE = 0.18215                      # rescales raw latents to roughly unit variance

@torch.no_grad()
def encode(image):                   # image: (B, 3, 512, 512) in [-1, 1]
    dist = vae.encode(image).latent_dist
    return dist.sample() * SCALE     # (B, 4, 64, 64): 48x fewer numbers

@torch.no_grad()
def decode(latents):
    return vae.decode(latents / SCALE).sample   # back to (B, 3, 512, 512)


# The diffusion loop never touches pixels. Training and sampling both
# happen on the (4, 64, 64) tensor; the VAE is frozen throughout.
def train_step_latent(unet, images, text_emb, scheduler):
    z0 = encode(images)
    t = torch.randint(0, 1000, (z0.size(0),), device=z0.device)
    noise = torch.randn_like(z0)
    z_t = scheduler.add_noise(z0, noise, t)
    return F.mse_loss(unet(z_t, t, text_emb).sample, noise)


# img2img is the same loop started part-way: encode, noise to t = strength * T, denoise.
def img2img(unet, image, text_emb, scheduler, strength=0.6, steps=30):
    z = encode(image)
    t_start = int(strength * 1000)
    z = scheduler.add_noise(z, torch.randn_like(z), torch.tensor([t_start]))
    for t in scheduler.timesteps_from(t_start, steps):
        z = scheduler.step(unet(z, t, text_emb).sample, t, z).prev_sample
    return decode(z)

The autoencoder has kept changing since 2022. Stable Diffusion 1 through SDXL use 4 latent channels; SD3 and FLUX moved to 16, which costs little in the denoiser and recovers most of the fine texture the 4-channel VAE lost. Some newer models trade the other way, with deeper compression (16× or 32× spatially) and more channels, so that the expensive transformer sees fewer tokens. In every case the split is the same: a cheap, frozen autoencoder that handles pixels, and an expensive denoiser that never has to.

A diffusion model conditioned on $c$ is just ${ϵ}_{\theta }(x_t,t,c)$: the same network with another input. For a class label, $c$ is an embedding added alongside the time embedding. For text, $c$ is a sequence of token embeddings from a frozen encoder (CLIP in Stable Diffusion 1 and 2, CLIP plus T5 in SD3 and FLUX), read through cross-attention layers so every spatial position can look at every word. Train with the condition present and the model learns $p(x\mid c)$.

In practice that alone gives weak prompt adherence. The model learns that the prompt is a factor, not the dominant one, and samples drift toward generic images. Dhariwal and Nichol first fixed this with an external classifier whose gradient pushed samples toward the class. Ho and Salimans then removed the classifier. Train a single model with the condition randomly dropped 10-20% of the time, so it can predict both $ϵ(x_t,t,c)$ and $ϵ(x_t,t,\varnothing )$. At sampling time, extrapolate:

The difference in brackets is the direction in which the condition changes the prediction. Scaling it by $w>1$ pushes harder in that direction than the model would on its own. Through the score-function lens this is sampling from $p(x\mid c)p(c\mid x{)}^{w-1}$, a distribution sharpened toward inputs the condition explains well. Stable Diffusion shipped with $w=7.5$. Higher values buy prompt fidelity with diversity, and past a point they buy saturation and artifacts. The cost is two network calls per step, which is why guidance-distilled models like FLUX.1-dev fold the scale in as an input and run one pass.

import torch

P_DROP = 0.1                                  # Ho & Salimans: drop the label 10-20% of the time

def train_step_cfg(model, x0, cond, schedule):
    """Same loss as before; the only change is occasionally hiding the condition."""
    bs = x0.size(0)
    drop = torch.rand(bs, device=x0.device) < P_DROP
    cond = torch.where(drop[:, None], NULL_EMBEDDING.expand_as(cond), cond)
    t = torch.randint(0, 1000, (bs,), device=x0.device)
    noise = torch.randn_like(x0)
    x_t = q_sample(x0, t, noise, schedule)
    return F.mse_loss(model(x_t, t, cond), noise)


@torch.no_grad()
def guided_eps(model, x_t, t, cond, w=7.5):
    """Two forward passes, one extrapolation. w = 1 is plain conditioning."""
    eps_cond   = model(x_t, t, cond)
    eps_uncond = model(x_t, t, NULL_EMBEDDING)
    return eps_uncond + w * (eps_cond - eps_uncond)

# In practice the two passes are batched: cat([x_t, x_t]) with cat([cond, null]),
# so guidance doubles the batch, not the number of kernel launches.
# A "negative prompt" simply replaces NULL_EMBEDDING with the embedding of
# text you want to move away from.

The 2D version is honest about a subtlety that image demos hide. Guidance does not move samples toward the “most typical” ring point; it moves them away from wherever the other classes sit. At moderate scales that sharpens the shape. Push $w$ to 5 or 8 and the ring bulges away from the spiral and the moons, with samples leaving the data entirely: the 2D version of oversaturation. On images the same mechanism amplifies whatever the model associates with the prompt relative to the unconditional average, which is where the characteristic “CFG look” comes from, and why the sweet spot is a model-specific number rather than “as high as possible”.

Look again at the closed form. The DDPM path from data to noise is $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$, with coefficients that sit on a quarter circle and a schedule that spends most of its steps near the noise end. The paths a trained model actually follows at sampling time inherit that shape: they curve, because at high noise every trajectory first heads for the average of the data and only later bends toward a specific sample. There is nothing sacred about any of this. Flow matching (Lipman et al.) and rectified flow (Liu et al.) ask for the simplest alternative: a straight line.

The network now predicts a velocity, the constant direction from the data point to its noise partner. Training is still one MSE at a random $t$. Sampling is Euler integration of the learned velocity field from $t=1$ back to $t=0$. The intuition usually offered is that straight training paths give straight sampling trajectories, which a few large Euler steps can follow. The toy below shows that this is only half the story. With data and noise paired at random, the learned trajectories still curve; on this toy they curve more than the DDPM model’s, because at high noise the velocity points at the data mean and every sample dives through the centre before heading out. What the flow model has instead is a well-conditioned target. The velocity is bounded everywhere, whereas recovering ${\hat{x}}_0$ from an $ϵ$ estimate at high noise divides by $\sqrt{{\bar{\alpha }}_t}\approx 0$ and amplifies every error, which is exactly what section 04 showed. Four Euler steps on the flow model beat four DDIM steps by a wide margin, and twenty match a thousand-step DDPM. Actually straightening the trajectories is a separate step: Liu et al. retrain on the (noise, sample) pairs the model itself produces, called reflow, and after a round or two the paths are straight enough for a single step.

The other half of the modern recipe is the denoiser. Peebles and Xie showed that a plain vision transformer on latent patches (DiT) scales more cleanly than a U-Net: no convolutions, no skip connections, just patch embedding, a stack of transformer blocks, and an unpatchify head. Timestep and conditioning enter through adaptive layer norm: an MLP turns the condition into a scale, a shift and a gate applied around each attention and MLP block, with the gates initialised to zero so every block starts as the identity. SD3 extended this to two streams, text tokens and image tokens with separate weights sharing one joint attention (MMDiT). FLUX uses that for its first blocks and then concatenates the streams. Every 2026 image and video model follows one of these two layouts.

import torch
import torch.nn.functional as F

def rectified_flow_train_step(model, x0, optimizer):
    """Same skeleton as DDPM training. Straight-line path, velocity target."""
    n = x0.size(0)
    t = torch.rand(n, device=x0.device)                      # continuous t in [0, 1]
    eps = torch.randn_like(x0)
    tt = t[:, None, None, None]
    x_t = (1 - tt) * x0 + tt * eps                            # linear interpolation
    v_target = eps - x0                                       # d x_t / d t, constant along the path
    loss = F.mse_loss(model(x_t, t), v_target)
    optimizer.zero_grad(); loss.backward(); optimizer.step()
    return loss.item()


@torch.no_grad()
def rectified_flow_sample(model, shape, steps=20, device="cpu"):
    """Euler integration from t = 1 (noise) to t = 0 (data)."""
    x = torch.randn(shape, device=device)
    dt = 1.0 / steps
    t = torch.ones(shape[0], device=device)
    for _ in range(steps):
        v = model(x, t)
        x = x - dt * v                                        # follow the velocity backwards
        t = t - dt
    return x

# SD3 samples t from a logit-normal instead of uniform, concentrating
# training on the middle of the path where the velocity is hardest to predict.

A well-conditioned target gets you to twenty steps. Getting to one to four needs either reflow or distillation: train a student to jump directly to where the teacher’s trajectory ends. Consistency models, LCM, adversarial diffusion distillation (SDXL Turbo), and FLUX.1-schnell are different ways of doing that, and every serious model now ships a full-quality checkpoint alongside a distilled few-step variant. The ideas from earlier sections all survive the transition: the closed-form jump is still a jump, guidance still works on velocities, and the latent autoencoder is unchanged.

A text-to-image system is five components, and by now every one of them has appeared on this page. A tokenizer and one or more frozen text encoders turn the prompt into a sequence of embeddings. A latent tensor is initialised with Gaussian noise. A denoiser, U-Net or DiT, is called once per step, twice if classifier-free guidance is on, and a scheduler turns its prediction into the next latent. After the last step a frozen VAE decoder turns the latent into pixels. Optionally a safety checker looks at the result.

The compute is almost entirely in the loop. For SD3 at 28 steps with guidance, that is 56 transformer forward passes on a 16×128×128 latent; the text encoders and the VAE decoder run once each and together cost less than two of those passes. That is why the sampler research of sections 08 and 11 translated so directly into product: halving the step count halves the bill.

import torch
from diffusers import StableDiffusion3Pipeline, FluxPipeline

# Everything on this page, wired together, is a one-liner in diffusers.
pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3.5-medium", torch_dtype=torch.bfloat16
).to("cuda")

image = pipe(
    prompt="a cosmos flower in a field at golden hour, macro photograph",
    negative_prompt="blurry, oversaturated",   # replaces the null branch of CFG
    guidance_scale=4.5,                        # w, section 10
    num_inference_steps=28,                    # Euler steps on the rectified flow, section 11
    height=1024, width=1024,                   # 16 x 128 x 128 latent, section 09
    generator=torch.Generator("cuda").manual_seed(0),   # fixes x_T, section 08
).images[0]

# Roughly what happens inside pipe(...):
#   text_emb  = text_encoders(tokenize(prompt))            # CLIP-L, CLIP-G, T5-XXL
#   z         = randn(1, 16, 128, 128)                     # x_T in latent space
#   for t in scheduler.timesteps:                          # 28 of them
#       v_c, v_u = transformer(cat([z, z]), t, cat([text_emb, null_emb])).chunk(2)
#       v        = v_u + w * (v_c - v_u)                   # classifier-free guidance
#       z        = scheduler.step(v, t, z).prev_sample     # one Euler step
#   image     = vae.decode(z / vae.config.scaling_factor)

# A distilled model skips guidance and most of the steps:
fast = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cuda")
image = fast(prompt="...", guidance_scale=0.0, num_inference_steps=4).images[0]

The names change every few months and the architecture diagrams keep getting busier, but the skeleton has been stable since 2022: compress, noise, learn to denoise conditioned on text, sample with an ODE solver, guide, decode. Video models add a time axis to the latent and otherwise run the same loop. Audio models run it on spectrograms. When a new model lands, the useful questions are the ones this page has been asking all along: what is the path, what does the network predict, how many steps, what does the autoencoder look like, and how is the condition injected.

Across these sections we followed one thread. A fixed process turns any image into Gaussian noise. Because that process folds into a single closed-form jump, a network can be trained on random timesteps with a plain squared error to predict the noise. Because the noise prediction is also a score, the reverse direction is a well-posed sampling problem, and once you see it as an ODE the step count falls from a thousand to twenty. Latent autoencoders shrink the problem, classifier-free guidance makes the prompt dominant, straight-line paths and transformers make it fast and scalable. None of these is the main idea. The systems that generate images today ship all of them at once, and the gap between the 2020 paper and a 2026 product is the composition.

Building this page changed how I hold the subject. I had read the equations before and could reproduce them, but I did not really believe the closed-form jump until I watched the chain and the one-step sample land on the same histogram, and I did not appreciate why guidance oversaturates until I saw the arrows pointing away from the other classes rather than toward the right one. Training a model in a browser tab, watching the loss stall at the entropy floor while the samples kept improving, taught me more about that loss curve than any number of plots in papers. The interactive form is not decoration here; it is how I checked that I understood.

I do not know what the next reformulation looks like. The path from noise to data keeps getting straighter and the step counts keep falling, the denoisers keep getting more transformer-shaped, and the boundary between “image model” and “video model” and “world model” keeps thinning. What seems stable is the set of questions: what is the corruption process, what does the network predict, how is the condition injected, how many evaluations does a sample cost. I will keep updating this page as the answers change, because they clearly have not stopped changing.