Evaluating generative vision: FID, CLIPScore, and human preference

1. Introduction: Why evaluating generated images is harder than it looks

Imagine two generative image models responding to the same prompt:

A small red delivery truck parked beside a blue bicycle on a rainy city street at night.

Model A produces a photorealistic street scene with beautiful reflections, convincing lighting, and impressive detail. Unfortunately, the bicycle is also red.

Model B produces a slightly less polished image but correctly renders the red truck, blue bicycle, and nighttime setting.

Which model is better?

The answer depends on what we measure.

• Visual quality: Does the image look natural, coherent, and free of obvious artifacts?

• Prompt adherence: Does it accurately represent the requested objects, colors, attributes, and relationships?

• Distributional similarity: Do its outputs resemble images from an appropriate reference distribution?

• Human preference: Which image would people actually choose?

• Diversity: Does the model produce varied outputs rather than repeating similar compositions?

These dimensions are related, but they are not interchangeable.

A model can improve visual realism while becoming worse at following instructions. It can produce images that people love while scoring poorly against a reference dataset. It can even improve an automated metric without producing a meaningful improvement in perceived quality.

This is why evaluating modern generative vision systems requires more than one number.

In this tutorial, we will study three complementary evaluation approaches:

1. Fréchet Inception Distance (FID) for comparing distributions of real and generated images.

2. CLIPScore and modern vision-language similarity metrics for estimating semantic alignment between generated images and prompts.

3. Human preference evaluation for determining which outputs people actually prefer.

We will also introduce Kernel Inception Distance (KID), build a reproducible Python evaluation pipeline using libraries released in 2025–2026, and examine a published experiment in which FID strongly disagreed with human reviewers.

Our implementation uses a 2025 vision-language encoder, SigLIP 2, with Transformers 5.x. We will distinguish the resulting modern similarity score from the original CLIPScore benchmark rather than misleadingly treating them as equivalent.

Finally, we will explore evaluation techniques specifically relevant to diffusion-generated images, including sampling steps, guidance strength, random seeds, modern multimodal judges, and compositional benchmarks.

By the end, you will understand not just how to calculate evaluation scores, but when to trust them, when to question them, and how to combine them into defensible evidence.

2. Three evaluation approaches at a glance

The first distinction is that these approaches answer different questions.

An additional category, learned preference scoring, attempts to approximate human judgments automatically.

Examples include PickScore, ImageReward, and HPS v2. These remain important historical baselines, but they should not be confused with actual human evaluation.

An automated preference predictor is another machine-learning model with its own training distribution, biases, and failure modes.

A relevant 2025 study, Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment, by Ying Ba and colleagues, examined this problem.

The researchers reported that preference models based on CLIP- and BLIP-derived representations, including PickScore and ImageReward, could penalize visually richer images that contained details extending beyond the literal prompt.

Their experiments investigated high-quality images, prompt optimization, and diffusion-model optimization, including work involving Stable Diffusion 3.5.

The study proposed complementary image-text alignment and aesthetic scoring components to address these shortcomings.

This does not mean every preference predictor fails on detailed images. It demonstrates that their rankings can depend on characteristics of the evaluation distribution, particularly when moving beyond the kinds of images represented in their training data.

The central evaluation principle

An evaluation method is useful only when the quantity it measures corresponds to the decision you need to make.

If you want to understand whether a generator produces realistic images across thousands of prompts, a distribution-level metric can help.

If your application must faithfully render products described by customers, semantic alignment and compositional correctness become essential.

If you are launching a consumer image-generation product, human preference and task-specific success rates may ultimately matter more than either automated score.

A trustworthy evaluation strategy measures these properties independently and investigates disagreements.

3. Understanding Fréchet Inception Distance

FID is one of the established distribution-level metrics for generative image evaluation.

Its underlying idea is straightforward:

Good generated images should have feature statistics similar to those of appropriate reference images.

However, the comparison happens in a learned feature space rather than directly over pixels.

3.1 How FID works

The conventional FID pipeline consists of four steps.

Step 1: Collect reference images.

These might be real product photographs, landscape images, or photographs from a held-out evaluation dataset.

Step 2: Generate comparison images.

Run your generative model against a predefined evaluation configuration and save its outputs.

Step 3: Extract image features.

Pass both image collections through a pretrained Inception-v3 feature extractor.

Conventional FID uses 2,048-dimensional features from the model’s final pooling representation.

Step 4: Compare feature distributions.

Estimate the mean vector and covariance matrix of each image collection, then calculate the Fréchet distance between the corresponding fitted multivariate Gaussian distributions.

The result is a scalar FID score.

3.2 The FID equation

Written using plain-text mathematical notation:

FID = ||μᵣ − μ𝗀||² + Tr(Σᵣ + Σ𝗀 − 2(Σᵣ¹ᐟ² Σ𝗀 Σᵣ¹ᐟ²)¹ᐟ²)

Where:

• μᵣ is the mean feature vector for reference images.

• μ𝗀 is the mean feature vector for generated images.

• Σᵣ is the covariance matrix of reference features.

• Σ𝗀 is the covariance matrix of generated features.

• Tr denotes the matrix trace.

• || · ||² represents squared Euclidean distance.

The first term measures how far apart the two feature-space centers are.

The covariance terms measure differences in how the features vary and correlate.

This matters because two datasets may have similar average features while displaying very different diversity patterns.

3.3 Interpreting FID

For a fixed evaluation protocol, lower is better.

A score of zero corresponds to identical fitted Gaussian feature distributions. Comparing an image collection against itself should therefore produce a result very close to zero, subject to numerical precision.

However, there is no universal FID threshold separating good images from bad images.

A score of 12 on one benchmark cannot automatically be interpreted as better than a score of 18 on another.

FID depends on:

• The reference dataset and its composition.

• The image preprocessing and resizing implementation.

• The feature extractor and selected feature dimension.

• The number of samples.

• The diversity of generated images.

• The statistical properties of both collections.

Comparisons are defensible only when these decisions are controlled.

3.4 Why FID is useful

FID offers several practical advantages.

It evaluates populations rather than isolated examples.

Instead of relying on one unusually impressive image, it compares statistics across an entire collection.

It captures more than pixel similarity.

Two photographs of the same general subject can have different pixels while occupying nearby regions of a learned feature space.

It can reveal distribution shifts.

Changes in color patterns, visual structures, textures, or category coverage may affect the resulting feature statistics.

It fits automated evaluation pipelines.

Once the reference dataset and preprocessing protocol are fixed, FID can be measured repeatedly throughout model development.

3.5 Where FID becomes misleading

Despite its usefulness, FID has several fundamental limitations.

Gaussian approximation

FID summarizes each feature distribution using a mean vector and covariance matrix.

Real image distributions may be multimodal and highly non-Gaussian.

Two distributions can share their first two moments while differing substantially in ways that FID cannot represent.

Limited instruction sensitivity

Inception features were not originally designed to verify modern text-to-image instructions.

A generator could create attractive photographs while consistently making mistakes involving object colors, counts, or spatial relationships.

FID might not penalize these errors adequately.

Sample-size sensitivity

Conventional FID requires estimating a covariance matrix over 2,048-dimensional features.

With only 120 images, the sample covariance has rank at most 119, making it necessarily singular.

This is not merely an abstract statistical concern. The fitted covariance matrix has thousands of directions without sufficient sample information.

Numerical square-root operations may become sensitive to implementation details, and even a finite computed FID can be highly biased.

Using a few hundred images does not solve the general problem.

For serious benchmark reporting, substantially larger collections, often tens of thousands of samples when feasible, are preferable.

Mode dropping

A generator might repeatedly produce similar-looking images while failing to represent important parts of the target distribution.

FID may detect some resulting distribution differences, but a low FID does not prove comprehensive mode coverage.

No prompt-level attribution

FID cannot directly identify:

• Which prompts failed.

• Whether object counts are correct.

• Whether attributes are assigned to the right objects.

• Whether requested text is legible.

• Whether a particular generated image is preferable to another.

Those questions require complementary evaluation methods.

4. Understanding CLIPScore and modern semantic alignment

FID compares collections of images.

CLIPScore evaluates the semantic relationship between images and text descriptions.

This distinction makes it useful for text-conditioned generation.

4.1 The original CLIPScore

CLIP uses separate image and text encoders that map visual and linguistic content into a shared representation space.

An image encoder produces an image embedding.

A text encoder produces a text embedding.

Semantically compatible images and descriptions tend to have higher similarity.

The commonly implemented CLIPScore formulation is:

CLIPScore(image, prompt) = max(100 × cosine(E_image, E_prompt), 0)

Here, cosine measures similarity between the two embeddings.

The result lies between 0 and 100.

Importantly, these values are not calibrated percentages of correctness.

A CLIPScore of 30 does not mean the generated image satisfies 30% of its prompt.

Scores should be compared using the same encoder, preprocessing pipeline, prompts, and model configuration.

The original OpenAI CLIP family remains relevant for reproducing historical benchmark results. However, its older checkpoints should not automatically be the default for a new evaluation system in 2026.

4.2 Moving to a 2025 vision-language encoder

For our modern evaluation example, we will use SigLIP 2, introduced in February 2025.

The model family is described in SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features.

SigLIP 2 provides:

• Image-text representations suitable for retrieval and similarity-based applications.

• Multilingual vision-language capabilities.

• Improvements in semantic understanding and localization relative to its predecessor.

• Checkpoints supporting different model sizes and resolution strategies.

We will use the documented checkpoint:

google/siglip2-base-patch16-224

Its inference interface is supported by the Transformers 5.x documentation.

There is an important naming distinction.

A cosine similarity score computed with SigLIP 2 is not the original CLIPScore benchmark.

Although we can reuse the same cosine-based numerical transformation, changing the encoder changes the metric.

We will call our implementation siglip2_similarity_x100.

This preserves the conceptual connection to CLIPScore without suggesting that scores obtained from different checkpoints are interchangeable.

4.3 Why semantic similarity is appealing

Unlike FID, text-image similarity does not require matching reference photographs.

Consider the prompt:

An origami spaceship floating above a city made of glass.

There may be no uniquely correct reference image for this fictional scene.

But a vision-language encoder can still compare the generated output with the textual description.

Semantic similarity can also be measured per prompt.

This makes it possible to investigate categories such as:

• Animals and everyday objects.

• Architecture and landscapes.

• Unusual materials.

• Product attributes.

• Artistic styles.

• Multi-object scenes.

Per-prompt evaluation makes these metrics valuable for diagnosing regressions.

4.4 The compositional correctness problem

Consider another prompt:

A yellow mug to the left of a purple notebook.

Now imagine an image containing a purple mug to the right of a yellow notebook.

Both objects are present.

Both colors are present.

The scene is broadly related to the prompt.

A global vision-language embedding might therefore produce relatively high similarity, even though the attribute assignments and spatial relationship are incorrect.

This is a central weakness of global semantic scoring.

A single similarity value does not necessarily verify every requested detail.

The 2025 benchmark T2I-CompBench++ directly addresses challenges such as attribute binding, object relationships, numeracy, and complex compositions. It contains 8,000 prompts organized into several categories.

For applications where precise relationships matter, global similarity should be supplemented with targeted correctness checks.

4.5 Another subtle limitation: prompt wording

Semantic scores depend on the text representation itself.

For instance, these prompts are closely related:

• A red bicycle beside a tree.

• A detailed photograph of a bright red bicycle positioned beside a leafy green tree.

• A bicycle near a tree.

They differ in specificity, visual style, and descriptive content.

A metric score may change because of the wording, even when the underlying image-generation behavior remains similar.

Therefore, never compare semantic alignment scores across changing prompt sets without accounting for those changes.

5. Human preference: Evaluating what people actually want

Human evaluation asks a different question.

Instead of comparing fitted statistical distributions or embedding similarities, we ask people to judge generated outputs directly.

For example:

Given the prompt and these two generated images, which one follows the instruction more accurately while maintaining acceptable visual quality?

This can become a rigorous evaluation task when the protocol is clearly defined.

5.1 Absolute ratings versus pairwise comparisons

An absolute evaluation might ask reviewers to rate an image from 1 to 5.

However, different reviewers interpret rating scales differently.

Pairwise comparison reduces some of this ambiguity.

A reviewer sees:

• The original prompt.

• Image on the left.

• Image on the right.

• A choice between left, right, or tie.

The model identities should remain hidden.

Presentation order should also be balanced to prevent systematic left/right effects.

5.2 Define the evaluation criterion

Human preference is not a single objective property.

Reviewers asked to select the most attractive image may tolerate incorrect prompt details.

Reviewers asked to prioritize instruction fidelity may choose a less polished but more accurate image.

Useful criteria include:

Visual quality

• Coherent geometry and anatomy.

• Convincing lighting and textures.

• Absence of rendering artifacts.

Prompt fidelity

• Required objects are present.

• Colors, quantities, and attributes are correct.

• Spatial relationships are correct.

• The requested setting and action are represented.

Aesthetic preference

• Composition and visual balance.

• Color harmony.

• Style and overall appeal.

For serious evaluation, collect separate judgments where these objectives might conflict.

5.3 Win rates and uncertainty

Suppose 200 prompts are evaluated.

For each prompt, reviewers compare outputs from Model A and Model B.

Imagine the following results:

Model A wins 62% of comparisons.

Model B wins 32%.

The remaining 6% are ties.

If ties receive half credit, Model A’s tie-adjusted preference score is:

(124 + 0.5 × 12) / 200 = 0.65

That is 65%.

However, this is still an estimate from a sample.

Its reliability depends on:

• Which prompts were selected.

• How outputs were generated.

• Which reviewers participated.

• The instructions given to reviewers.

• Whether judgments were genuinely independent.

Confidence intervals and sampling details are essential when interpreting human preference results.

5.4 Why reviewer-level independence matters

Suppose five reviewers assess the same generated image pair.

Those five judgments share the same prompt and visual content.

Treating them as five entirely independent prompts would exaggerate the effective sample size.

A better approach is to aggregate judgments within each prompt and then calculate uncertainty across prompts.

Later, we will implement a prompt-level bootstrap to handle this structure.

6. Evaluating diffusion-generated images

Diffusion models deserve special consideration because image quality depends not only on the trained model but also on the sampling procedure.

A diffusion-generation system typically starts from noise and iteratively constructs an image through a denoising process.

Different inference configurations can produce noticeably different outputs from the same model checkpoint.

Consequently, a comparison between diffusion systems must specify more than the model name.

6.1 Sampling steps influence evaluation

The number of inference steps affects the computational process used to generate an image.

For a fixed scheduler and model configuration, more steps may improve reconstruction or detail up to a point.

However, improvement is not guaranteed to be monotonic.

Modern samplers can generate strong results with relatively few steps, and their optimal step counts depend on the model and training procedure.

A benchmark that compares one system using four inference steps against another using fifty steps is comparing different quality-cost configurations.

That may be appropriate for a product decision, but it is not a controlled comparison of model architectures alone.

A useful diffusion evaluation reports both:

• Quality metrics under a defined sampling configuration.

• Runtime or compute cost per generated image.

6.2 Guidance strength changes the output distribution

Classifier-free guidance, where supported, adjusts the influence of text conditioning during generation.

Changing guidance strength can affect:

• Prompt adherence.

• Image diversity.

• Color saturation.

• Detail and visual artifacts.

• Perceived realism.

Increasing guidance does not necessarily improve every evaluation dimension.

A configuration may achieve stronger text-image similarity while reducing diversity or producing unnatural visual characteristics.

Different diffusion architectures also use different guidance conventions.

Some models use explicit classifier-free guidance, while others use distilled or model-specific guidance mechanisms.

Therefore, guidance values are not universally comparable across model families.

6.3 Random seeds and stochastic variation

Diffusion outputs can vary with the initial noise and sampling configuration.

Evaluating one output per prompt measures only one realization of the generator’s behavior.

Suppose a model produces excellent images for a prompt under some seeds but fails under others.

A single successful example cannot establish reliability.

For robust evaluation:

1. Use a predefined seed schedule.

2. Generate multiple samples per prompt when feasible.

3. Avoid choosing only the best-looking output.

4. Record the number of samples and any selection policy.

5. Use prompt-level aggregation when reporting results.

Matching seed integers across different diffusion architectures does not guarantee equivalent initial noise or equivalent random trajectories.

The comparison remains useful as a controlled experimental convention, but it should not be mistaken for perfect stochastic matching.

6.4 A practical diffusion evaluation matrix

A controlled experiment can vary one generation parameter at a time.

For each configuration, collect several metrics rather than relying on a single score.

A production diffusion evaluation should combine:

• Distributional measurements.

• Prompt-conditioned semantic similarity.

• Compositional correctness.

• Human preference.

• Generation latency and resource usage.

The best configuration is the one that meets the application’s quality requirements at an acceptable operational cost.

7. Building a reproducible evaluation environment

We will now implement a practical evaluation pipeline.

Our components are:

• PyTorch for inference and tensor operations.

• TorchMetrics for FID and KID.

• Torch-fidelity as the supported Inception feature-extraction dependency.

• Transformers 5.x for SigLIP 2 inference.

• Pillow for image processing.

• NumPy for numerical operations and bootstrap analysis.

• SciPy as part of the numerical environment.

7.1 Why we are not using the older Clean-FID release

Clean-FID contributed important work on standardized image preprocessing.

However, its last PyPI release, version 0.1.35, dates to December 2022.

For a modern tutorial requiring recently released dependencies, we will instead use TorchMetrics 1.9.0, released in March 2026, together with torch-fidelity 0.4.0, released in February 2026.

TorchMetrics provides documented interfaces for Fréchet Inception Distance and Kernel Inception Distance.

The two metrics can be updated incrementally using image batches, making them suitable for reusable evaluation pipelines.

The exact implementation still matters: different FID implementations can produce different numerical results because of preprocessing, feature extraction, and statistical conventions.

7.2 Python and dependency versions

Use CPython 3.12 (any 3.12.x patch release) for this example.

We will pin the direct dependencies instead of installing unrestricted latest versions.

Create a file named requirements.txt.

torch==2.10.0
torchvision==0.25.0
torchmetrics==1.9.0
torch-fidelity==0.4.0
transformers==5.20.0
numpy==2.3.5
scipy==1.17.1
Pillow==12.1.1

These are specific published package versions from the 2025–2026 period, including the matching PyTorch and TorchVision release pair and the current major Transformers generation.

The code below uses their documented interfaces. Because installation binaries and accelerator backends vary across operating systems, the package combination should still be smoke-tested in the target environment before being designated a production-validated lockfile.

For exact long-term reproducibility, additionally lock transitive dependencies and artifact hashes for your deployment platform.

7.3 Create the virtual environment

On Linux or macOS:

python3.12 -m venv .venv
source .venv/bin/activate

On Windows PowerShell:

py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1

Verify the interpreter version.

python --version

Install the pinned dependencies.

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip check

For accelerator-specific PyTorch distributions, consult the official PyTorch installation documentation and ensure the resulting Torch and TorchVision versions remain compatible.

7.4 Verify the installed environment

Save the following as verify_env.py.

import sys
from importlib.metadata import version

required = {
    "torch": "2.10.0",
    "torchvision": "0.25.0",
    "torchmetrics": "1.9.0",
    "torch-fidelity": "0.4.0",
    "transformers": "5.20.0",
    "numpy": "2.3.5",
    "scipy": "1.17.1",
    "Pillow": "12.1.1",
}

if sys.version_info[:2] != (3, 12):
    raise RuntimeError(
        "Expected CPython 3.12.x; "
        f"found {sys.version.split()[0]}"
    )

for package, expected in required.items():
    actual = version(package)

    # PyTorch CPU/CUDA wheels carry a local tag such as "2.10.0+cpu".
    if actual.split("+")[0] != expected:
        raise RuntimeError(
            f"{package}: expected {expected}, "
            f"found {actual}"
        )

    print(f"{package}: {actual}")

import torch
from torchmetrics.image.fid import FrechetInceptionDistance
from torchmetrics.image.kid import KernelInceptionDistance
from transformers import AutoModel, AutoProcessor

print(f"CUDA available: {torch.cuda.is_available()}")
print("Environment verification passed")

Run:

python verify_env.py

This checks interpreter and package versions and verifies that the key metric and model-loading interfaces can be imported.

It does not validate downloaded model weights or complete metric execution; those are covered by the end-to-end example.

Finally, save the resolved environment:

python -m pip freeze > requirements-resolved.txt

This records the full dependency resolution for your machine.

For reproducible deployment, convert that resolution into a platform-specific lockfile with hashes, and retain the model checkpoint revision used for scoring.

7.5 Organize the project

Our project will use this structure:

vision-evaluation/
├── requirements.txt
├── requirements-resolved.txt
├── verify_env.py
├── make_demo_data.py
├── evaluate.py
├── prepare_human.py
├── summarize_human.py
└── evaluation/
    ├── prompts.csv
    ├── reference/
    ├── candidate_a/
    ├── candidate_b/
    ├── public/
    └── private/

Each candidate folder will contain images with the same filenames.

The manifest associates each image identifier with its prompt.

FID and KID compare image collections without considering prompt assignments.

Our modern semantic similarity score uses those assignments explicitly.

That difference is the foundation of the experiment.

8. Create a controlled synthetic benchmark

Rather than downloading a potentially inappropriate benchmark, we will generate a small neutral image collection.

Each image contains a colored geometric shape against a light-gray background.

Prompts explicitly describe the shapes and colors.

For example:

A flat illustration of one green triangle on a light gray background.

We will create three image collections:

Reference: Correctly labeled geometric images.

Candidate A: Images satisfying their associated prompts, with different positions and sizes from the reference.

Candidate B: Exactly the same image collection as Candidate A, but assigned to different prompts.

Candidate B will deliberately mismatch colors.

This creates a controlled experiment in which both candidate image distributions are identical, yet their prompt-image relationships differ.

8.1 Dataset generator

Save this script as make_demo_data.py.

import csv
import random
import shutil
from pathlib import Path

from PIL import Image, ImageDraw

root = Path("evaluation")

folders = (
    "reference",
    "candidate_a",
    "candidate_b",
)

for name in folders:
    folder = root / name

    if folder.exists():
        shutil.rmtree(folder)

    folder.mkdir(parents=True)

colors = {
    "red": (220, 62, 57),
    "blue": (55, 113, 217),
    "green": (42, 157, 100),
    "orange": (245, 148, 45),
}

shapes = ("circle", "square", "triangle")
count = 120


def render(color, shape, seed, destination):
    rng = random.Random(seed)

    image = Image.new(
        "RGB",
        (256, 256),
        (238, 238, 238),
    )

    draw = ImageDraw.Draw(image)

    cx = 128 + rng.randint(-14, 14)
    cy = 128 + rng.randint(-14, 14)
    radius = rng.randint(62, 80)

    box = (
        cx - radius,
        cy - radius,
        cx + radius,
        cy + radius,
    )

    if shape == "circle":
        draw.ellipse(box, fill=color)

    elif shape == "square":
        draw.rectangle(box, fill=color)

    elif shape == "triangle":
        points = [
            (cx, cy - radius),
            (cx - radius, cy + radius),
            (cx + radius, cy + radius),
        ]

        draw.polygon(points, fill=color)

    image.save(destination)


with (root / "prompts.csv").open(
    "w",
    newline="",
    encoding="utf-8",
) as handle:
    writer = csv.DictWriter(
        handle,
        fieldnames=["image_id", "prompt"],
    )

    writer.writeheader()

    for i in range(count):
        image_id = f"{i:04d}.png"

        color_name = list(colors)[
            i % len(colors)
        ]

        shape = shapes[
            (i // len(colors)) % len(shapes)
        ]

        prompt = (
            f"A flat illustration of one "
            f"{color_name} {shape} "
            f"on a light gray background."
        )

        writer.writerow({
            "image_id": image_id,
            "prompt": prompt,
        })

        render(
            colors[color_name],
            shape,
            1000 + i,
            root / "reference" / image_id,
        )

        render(
            colors[color_name],
            shape,
            2000 + i,
            root / "candidate_a" / image_id,
        )


# Candidate B contains exactly the same image
# multiset as A, with incorrect prompt assignments.

for i in range(count):
    source_id = f"{(i + 1) % count:04d}.png"
    target_id = f"{i:04d}.png"

    shutil.copyfile(
        root / "candidate_a" / source_id,
        root / "candidate_b" / target_id,
    )

print(
    f"Created {count} prompts and "
    "three image collections."
)

Execute the generator.

python make_demo_data.py

The resulting collections each contain 120 images.

Candidate A and Candidate B contain precisely the same underlying image files, just with different filename assignments.

Because prompt colors cycle through four possibilities, shifting the images by one position introduces a systematic color mismatch.

8.2 What should happen?

The expected relationship is:

Notice the distinction between mathematical expectation and empirical prediction.

The distributional equality follows from the dataset construction.

Whether the vision-language encoder reliably detects these simple color differences is an empirical question.

The encoder could assign similar scores to both candidates, especially because minimal graphics differ from its richer pretraining examples.

That would not invalidate the experiment.

It would reveal a limitation of that particular semantic evaluator.

9. Numerical stability: Why we need KID alongside FID

Before writing the scoring script, we must address an important statistical issue.

Our synthetic benchmark contains only 120 images per collection.

Conventional FID uses 2,048-dimensional image features.

With 120 samples, the estimated covariance matrix has rank at most 119.

Consequently, the conventional covariance estimate is rank-deficient.

Even when an implementation can compute a finite result, that result is not automatically statistically reliable.

9.1 Use a reduced-feature FID diagnostic

TorchMetrics supports several Inception feature dimensions.

For this small experiment, we will use 64-dimensional features.

This removes the unavoidable dimensionality-versus-sample-count singularity found with 2,048 features and 120 samples.

However, the covariance may still be singular or poorly conditioned if the feature vectors occupy a low-dimensional region.

Reduced feature dimension does not guarantee full rank.

We will label the result explicitly as FID-64, because it is not numerically comparable to conventional FID-2048.

Its purpose is to demonstrate the distributional invariance property.

For a publication benchmark, use the conventional feature configuration and a suitably large dataset.

9.2 Add Kernel Inception Distance

KID uses a kernel-based maximum mean discrepancy instead of fitting Gaussian distributions.

Its conventional form uses a polynomial kernel on Inception features.

Unlike the plug-in FID estimator, the usual KID statistic is an unbiased estimator of squared MMD when its sampling assumptions hold.

KID does not require computing a matrix square root of an estimated covariance matrix.

This makes it useful for smaller datasets.

However, KID still has sampling variability and depends on its feature representation.

It is not a guarantee of accurate perceptual evaluation.

9.3 Our small-sample configuration

For the demonstration:

For KID, we will use the full 120-image collection as one subset.

This makes the computation invariant to image reordering, apart from floating-point effects.

It also means that the within-run subset standard deviation is not meaningful as an uncertainty interval.

For a larger benchmark, we would use repeated subsets or independent samples and report uncertainty appropriately.

10. Calculate FID, KID, and modern semantic similarity

We are ready to build the scoring pipeline.

10.1 Loading images correctly

TorchMetrics’ standard Inception-based FID and KID implementations accept image tensors with:

• Shape (batch_size, 3, height, width).

• Unsigned 8-bit pixel values.

• Pixel intensities from 0 to 255.

The metric handles Inception-specific processing internally.

SigLIP 2 uses its own processor for image and text inputs.

We should not manually reuse Inception preprocessing for SigLIP 2.

10.2 Complete evaluation script

Save the following as evaluate.py.

import csv
import json
import math
import os
from pathlib import Path

import numpy as np
import torch

from PIL import Image
from torchmetrics.image.fid import FrechetInceptionDistance
from torchmetrics.image.kid import KernelInceptionDistance
from transformers import AutoModel, AutoProcessor

root = Path("evaluation")
batch_size = 8

model_id = "google/siglip2-base-patch16-224"

# For a published experiment, set this environment
# variable to an immutable checkpoint commit hash.
revision = os.environ.get(
    "VISION_MODEL_REVISION",
    "main",
)

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

torch.manual_seed(2026)
np.random.seed(2026)

with (root / "prompts.csv").open(
    newline="",
    encoding="utf-8",
) as handle:
    rows = list(csv.DictReader(handle))

image_ids = [row["image_id"] for row in rows]

if not rows or len(set(image_ids)) != len(rows):
    raise ValueError(
        "The manifest must contain unique image IDs."
    )

expected = set(image_ids)

for name in (
    "reference",
    "candidate_a",
    "candidate_b",
):
    actual = {
        path.name
        for path in (root / name).glob("*.png")
    }

    if actual != expected:
        raise ValueError(
            f"{name}: filenames do not match manifest."
        )


def load_tensor(path):
    with Image.open(path) as image:
        array = np.array(
            image.convert("RGB"),
            dtype=np.uint8,
            copy=True,
        )

    return torch.from_numpy(array).permute(2, 0, 1)


def image_batches(folder):
    for start in range(0, len(rows), batch_size):
        batch_rows = rows[start:start + batch_size]

        images = [
            load_tensor(folder / row["image_id"])
            for row in batch_rows
        ]

        yield torch.stack(images).to(device)


fid_metric = FrechetInceptionDistance(
    feature=64,
    reset_real_features=False,
    normalize=False,
).to(device)

kid_metric = KernelInceptionDistance(
    feature=2048,
    subsets=1,
    subset_size=len(rows),
    reset_real_features=False,
    normalize=False,
).to(device)

fid_metric.eval()
kid_metric.eval()

# Extract reference features only once.
with torch.inference_mode():
    for images in image_batches(root / "reference"):
        fid_metric.update(images, real=True)
        kid_metric.update(images, real=True)

results = {}

for name in ("candidate_a", "candidate_b"):
    folder = root / name

    with torch.inference_mode():
        for images in image_batches(folder):
            fid_metric.update(images, real=False)
            kid_metric.update(images, real=False)

    fid_value = float(fid_metric.compute().item())

    kid_mean, kid_std = kid_metric.compute()

    results[name] = {
        "fid64": fid_value,
        "kid2048": float(kid_mean.item()),
    }

    # Preserve reference statistics for the next model.
    fid_metric.reset()
    kid_metric.reset()

# Modern text-image similarity.
processor = AutoProcessor.from_pretrained(
    model_id,
    revision=revision,
)

model = AutoModel.from_pretrained(
    model_id,
    revision=revision,
).to(device).eval()

per_prompt = []

for name in ("candidate_a", "candidate_b"):
    folder = root / name
    scores = []

    with torch.inference_mode():
        for start in range(0, len(rows), batch_size):
            batch_rows = rows[start:start + batch_size]

            images = [
                Image.open(
                    folder / row["image_id"]
                ).convert("RGB")
                for row in batch_rows
            ]

            prompts = [
                row["prompt"]
                for row in batch_rows
            ]

            inputs = processor(
                text=prompts,
                images=images,
                padding="max_length",
                max_length=64,
                return_tensors="pt",
            ).to(device)

            outputs = model(**inputs)

            similarities = torch.nn.functional.cosine_similarity(
                outputs.image_embeds,
                outputs.text_embeds,
                dim=-1,
            )

            batch_scores = (
                similarities.clamp(min=0) * 100
            ).cpu().tolist()

            for row, score in zip(
                batch_rows,
                batch_scores,
            ):
                per_prompt.append({
                    "candidate": name,
                    "image_id": row["image_id"],
                    "siglip2_similarity_x100": score,
                })

            scores.extend(batch_scores)

            for image in images:
                image.close()

    results[name]["siglip2_similarity_x100"] = float(
        np.mean(scores)
    )

    results[name]["sample_count"] = len(rows)

# The image distributions are identical.
# Large differences would indicate an implementation
# or numerical-stability issue.
for metric_name, tolerance in (
    ("fid64", 0.02),
    ("kid2048", 0.001),
):
    a = results["candidate_a"][metric_name]
    b = results["candidate_b"][metric_name]

    if not math.isfinite(a) or not math.isfinite(b):
        raise RuntimeError(
            f"Non-finite result for {metric_name}"
        )

    if abs(a - b) > tolerance:
        raise RuntimeError(
            f"{metric_name} differs unexpectedly: "
            f"{a} versus {b}"
        )

with (root / "per_prompt_scores.csv").open(
    "w",
    newline="",
    encoding="utf-8",
) as handle:
    writer = csv.DictWriter(
        handle,
        fieldnames=[
            "candidate",
            "image_id",
            "siglip2_similarity_x100",
        ],
    )

    writer.writeheader()
    writer.writerows(per_prompt)

results["_configuration"] = {
    "device": str(device),
    "fid_feature_dimension": 64,
    "kid_feature_dimension": 2048,
    "kid_subset_size": len(rows),
    "semantic_encoder": model_id,
    "requested_revision": revision,
}

with (root / "results.json").open(
    "w",
    encoding="utf-8",
) as handle:
    json.dump(results, handle, indent=2)

print(json.dumps(results, indent=2))

Execute it:

python evaluate.py

The first run downloads the relevant pretrained weights.

A compatible GPU can accelerate feature extraction, although CPU execution is supported.

10.3 Understanding the implementation

Several details deserve attention.

Explicit feature dimensions

We instantiate FID with 64-dimensional features.

This is intentional for the small synthetic demonstration.

We also calculate a 2,048-dimensional KID, which provides a more appropriate small-sample distributional comparison without requiring covariance matrix square roots.

Reference-feature caching

Both TorchMetrics objects use reset_real_features=False.

After each candidate is evaluated, we reset the accumulated generated-image features while retaining the reference features.

This avoids unnecessarily extracting reference features for every candidate.

Independent preprocessing

TorchMetrics handles Inception image processing.

SigLIP 2 receives images through its own AutoProcessor.

For SigLIP 2 text, we use maximum-length padding because this matches the documented model input convention.

Inference mode

The script uses torch.inference_mode().

We are evaluating pretrained models rather than updating their weights, so gradient tracking is unnecessary.

Per-prompt scores

The script creates per_prompt_scores.csv.

This allows us to investigate individual failures rather than relying exclusively on the average.

A model could have a strong overall similarity score while consistently failing on a particular category.

Numerical checks

The script verifies that the two distributional scores are close within a small numerical tolerance.

Because Candidate A and Candidate B contain the same image multiset, their distributional scores should agree.

This is a useful functional validation of the experiment.

The numerical tolerances are diagnostic rather than universal acceptance thresholds for metric implementations.

10.4 Expected result structure

The output contains the following measurements for each candidate:

The output also records the device, feature dimensions, model identifier, and requested checkpoint revision.

We deliberately do not prescribe exact numerical values.

The model weights, hardware environment, and floating-point execution path affect the resulting measurements.

The important assertions are:

• FID-64 should be approximately identical for both candidates.

• KID-2048 should be approximately identical for both candidates.

• Candidate A is semantically correct by construction.

• Whether SigLIP 2 detects that advantage must be measured.

For reference, here is what we measured on a CPU-only Linux machine (Python 3.12, the pinned versions above, SigLIP 2 at revision main, about 2.5 minutes end to end):

Both distribution metrics are identical, exactly as the construction predicts. KID is slightly negative because it is an unbiased estimator: for two sets drawn from the same distribution its expected value is zero, so small negative values are normal noise, not a bug. SigLIP 2 did detect the color mismatch, scoring the correct candidate about 5 points higher. Your numbers can differ slightly with other hardware or checkpoint revisions; the pattern should not.

10.5 Interpreting semantic scores

Our SigLIP 2 implementation calculates:

SigLIP2 similarity = max(100 × cosine(image_embedding, text_embedding), 0)

This resembles the CLIPScore mathematical transformation.

However, it is not the original CLIPScore benchmark.

Furthermore, SigLIP 2 was trained with a sigmoid-based image-text objective.

The cosine transformation does not turn its output into a calibrated probability.

We are using it as a reproducible relative similarity measure.

SigLIP 2 also exposes model-specific image-text logits, but those logits should not be interchanged numerically with cosine similarity.

Choose one scoring definition, document it, and use it consistently.

10.6 Check the actual checkpoint revision

Pinning Python dependencies is not sufficient if the pretrained model checkpoint can change.

Our example permits setting an immutable Hugging Face revision.

Once you have selected a specific model commit, run:

export VISION_MODEL_REVISION="YOUR_VERIFIED_COMMIT_HASH"
python evaluate.py

Replace the placeholder with an actual commit identifier from the model repository.

For publication, record that identifier alongside the package lockfile and output results.

Do not invent or approximate a commit hash.

The processor and model should be loaded from the same revision.

11. What the synthetic experiment teaches us

The most important result is mathematical rather than empirical.

Candidate A and Candidate B contain identical image multisets.

Their marginal image distributions are therefore identical.

Any properly permutation-invariant distributional metric must assign them the same result against a fixed reference distribution, subject to numerical effects.

However, Candidate B maps images to incorrect prompts.

This means that prompt-conditioned correctness differs even though the marginal image distribution does not.

This distinction can be expressed conceptually as:

• Distributional evaluation examines P(image).

• Conditional generation quality depends on P(image | prompt).

A model can reproduce the correct collection of images while assigning them to the wrong instructions.

FID and KID do not use prompt assignments.

They cannot detect this specific failure.

A semantic alignment metric has an opportunity to detect it because prompts are included in the evaluation.

Human reviewers can verify the intended attribute mismatch directly.

The example is intentionally small and artificial.

It is not evidence that any particular production text-to-image model is superior.

Its value is that it isolates a metric limitation that can otherwise be difficult to observe.

12. Implementing a blinded human preference study

Automated scores are only part of the evaluation.

Next, we will prepare a human comparison protocol using the same generated images.

Our objective is to determine whether reviewers prefer Candidate A when asked to prioritize prompt fidelity.

12.1 Why blinding and counterbalancing matter

Suppose a reviewer knows that the left image came from an expensive commercial model and the right image came from an experimental system.

That knowledge may influence judgment.

A proper comparison should hide model names.

Left/right assignment must also be balanced.

If every reviewer sees the same model on the left for a particular prompt, position effects and prompt effects can become entangled.

Our revised implementation will create reviewer-specific image assignments.

Each reviewer receives:

• A unique reviewer identifier.

• A set of comparison trials.

• Anonymous left/right images.

• The original prompt.

• A response field for left, right, or tie.

The private mapping from anonymous images to candidate identities is stored separately from reviewer-accessible materials.

12.2 Prepare the blinded comparisons

Save the following as prepare_human.py.

import csv
import shutil
from pathlib import Path

root = Path("evaluation")

public = root / "public"
private = root / "private"

for directory in (public, private):
    if directory.exists():
        shutil.rmtree(directory)

    directory.mkdir(parents=True)

with (root / "prompts.csv").open(
    newline="",
    encoding="utf-8",
) as handle:
    rows = list(csv.DictReader(handle))

reviewers = ["R01", "R02", "R03"]

trials_path = public / "trials.csv"
key_path = private / "key.csv"
votes_path = public / "votes_template.csv"

with (
    trials_path.open(
        "w", newline="", encoding="utf-8"
    ) as trials_file,
    key_path.open(
        "w", newline="", encoding="utf-8"
    ) as key_file,
    votes_path.open(
        "w", newline="", encoding="utf-8"
    ) as votes_file,
):
    trials = csv.writer(trials_file)
    key = csv.writer(key_file)
    votes = csv.writer(votes_file)

    trials.writerow([
        "reviewer_id",
        "trial_id",
        "prompt",
        "left_image",
        "right_image",
    ])

    key.writerow([
        "trial_id",
        "reviewer_id",
        "image_id",
        "left_model",
        "right_model",
    ])

    votes.writerow([
        "reviewer_id",
        "trial_id",
        "choice",
    ])

    for i, row in enumerate(rows):
        image_id = row["image_id"]
        stem = Path(image_id).stem

        for j, reviewer_id in enumerate(reviewers):
            trial_id = f"{reviewer_id}_{stem}"

            # Balanced across reviewers and prompts.
            left_is_a = (i + j) % 2 == 0

            if left_is_a:
                left_model = "candidate_a"
                right_model = "candidate_b"
            else:
                left_model = "candidate_b"
                right_model = "candidate_a"

            image_dir = (
                public / "images" / reviewer_id
            )
            image_dir.mkdir(
                parents=True,
                exist_ok=True,
            )

            left_name = f"{stem}_left.png"
            right_name = f"{stem}_right.png"

            shutil.copyfile(
                root / left_model / image_id,
                image_dir / left_name,
            )

            shutil.copyfile(
                root / right_model / image_id,
                image_dir / right_name,
            )

            relative_left = (
                f"images/{reviewer_id}/{left_name}"
            )
            relative_right = (
                f"images/{reviewer_id}/{right_name}"
            )

            trials.writerow([
                reviewer_id,
                trial_id,
                row["prompt"],
                relative_left,
                relative_right,
            ])

            key.writerow([
                trial_id,
                reviewer_id,
                image_id,
                left_model,
                right_model,
            ])

            votes.writerow([
                reviewer_id,
                trial_id,
                "",
            ])

print(
    f"Prepared {len(rows)} prompts for "
    f"{len(reviewers)} reviewers."
)

print(f"Public materials: {public}")
print(f"Private model key: {key_path}")

Run:

python prepare_human.py

This creates 360 comparison trials across three reviewers.

Each reviewer receives 120 prompts.

For every reviewer, Candidate A appears on the left for half the prompts and on the right for the other half.

Across reviewers, the assignments alternate for each prompt.

For an odd number of reviewers, perfect per-prompt balance is impossible, but the design prevents the same fixed left/right assignment from being shown to everyone.

12.3 Keep identities private

The directory separation is important:

evaluation/
├── public/
│   ├── trials.csv
│   ├── votes_template.csv
│   └── images/
│       ├── R01/
│       ├── R02/
│       └── R03/
└── private/
    └── key.csv

Only the public materials should be distributed to reviewers.

The private key must not be available through the review interface or shared file directory.

Simply naming a folder private is not an access-control mechanism. A deployed study must enforce the separation through actual permissions or isolated storage.

12.4 Collect responses

For the synthetic experiment, reviewers should answer:

Which image more accurately represents the requested color and geometric shape?

The allowed choices are:

• left

• right

• tie

Create the working response file.

cp evaluation/public/votes_template.csv evaluation/public/votes.csv

On Windows PowerShell, use:

Copy-Item evaluation/public/votes_template.csv evaluation/public/votes.csv

Enter reviewer responses in the choice column.

Keep each reviewer’s identifier and trial identifier unchanged.

In a production interface, responses would normally be collected through a form or annotation system rather than by manually editing CSV files.

The CSV workflow is sufficient for a reproducible demonstration.

13. Calculate human preference and confidence intervals

We now need to translate anonymous left/right choices back into model preferences.

We will also estimate statistical uncertainty using a prompt-level bootstrap.

13.1 Why bootstrap prompts instead of votes?

Suppose three reviewers independently assess the same image pair.

Their judgments may differ, but all three depend on the same visual content.

A naive bootstrap that treats every vote as an unrelated observation can underestimate uncertainty.

Instead, we will first aggregate judgments within each prompt.

Then we will resample prompts.

This estimates variability across the sampled prompt set while retaining the within-prompt aggregation.

13.2 Complete preference summary script

Save as summarize_human.py.

import csv
from collections import Counter, defaultdict
from pathlib import Path

import numpy as np

root = Path("evaluation")

key_path = root / "private" / "key.csv"
votes_path = root / "public" / "votes.csv"

with key_path.open(
    newline="",
    encoding="utf-8",
) as handle:
    keys = {
        row["trial_id"]: row
        for row in csv.DictReader(handle)
    }

scores_by_prompt = defaultdict(list)
outcomes = Counter()
seen_trials = set()

with votes_path.open(
    newline="",
    encoding="utf-8",
) as handle:
    for row in csv.DictReader(handle):
        choice = row["choice"].strip().lower()

        if not choice:
            continue

        trial_id = row["trial_id"]
        reviewer_id = row["reviewer_id"]

        if trial_id not in keys:
            raise ValueError(
                f"Unknown trial: {trial_id}"
            )

        if trial_id in seen_trials:
            raise ValueError(
                f"Duplicate response: {trial_id}"
            )

        if reviewer_id != keys[trial_id]["reviewer_id"]:
            raise ValueError(
                f"Reviewer mismatch: {trial_id}"
            )

        if choice not in ("left", "right", "tie"):
            raise ValueError(
                f"Invalid choice: {choice}"
            )

        seen_trials.add(trial_id)

        image_id = keys[trial_id]["image_id"]

        if choice == "tie":
            outcome = "tie"
            score = 0.5
        else:
            outcome = keys[trial_id][
                f"{choice}_model"
            ]

            score = (
                1.0
                if outcome == "candidate_a"
                else 0.0
            )

        outcomes[outcome] += 1
        scores_by_prompt[image_id].append(score)

if not scores_by_prompt:
    raise ValueError(
        "No completed votes were found."
    )

prompt_scores = np.array(
    [
        np.mean(scores)
        for scores in scores_by_prompt.values()
    ],
    dtype=np.float64,
)

rng = np.random.default_rng(2026)

bootstrap_means = np.empty(
    5000,
    dtype=np.float64,
)

for i in range(5000):
    sampled = rng.choice(
        prompt_scores,
        size=len(prompt_scores),
        replace=True,
    )

    bootstrap_means[i] = sampled.mean()

low, high = np.quantile(
    bootstrap_means,
    [0.025, 0.975],
)

print(
    f"Prompts assessed: {len(prompt_scores)}"
)

print(
    f"Individual votes: {sum(outcomes.values())}"
)

print(
    f"Raw outcomes: {dict(outcomes)}"
)

print(
    f"Candidate A tie-adjusted preference: "
    f"{prompt_scores.mean():.1%}"
)

print(
    f"95% prompt-bootstrap interval: "
    f"[{low:.1%}, {high:.1%}]"
)

Run:

python summarize_human.py

The program reports:

• Number of unique prompts assessed.

• Total reviewer judgments.

• Raw candidate wins and ties.

• Candidate A’s tie-adjusted preference score.

• A 95% prompt-bootstrap interval.

13.3 Statistical interpretation

This bootstrap is useful, but it has assumptions.

It treats the sampled prompts as representative units of the intended task distribution.

It does not automatically generalize to all possible prompts or reviewer populations.

Reviewers in this demonstration are a small fixed group.

A larger study should consider reviewer recruitment, randomized trial order, repeated comparisons, and reviewer-level effects.

If the target quantity is an average over a broader reviewer population, a hierarchical approach that accounts for both prompts and reviewers may be more appropriate.

For model release decisions, the protocol and uncertainty method should be selected before looking at final results.

13.4 What we should expect

Candidate A correctly represents the requested color and shape.

Candidate B systematically mismatches the colors.

Attentive reviewers prioritizing prompt fidelity should therefore favor Candidate A.

The exact preference score remains empirical.

The important comparison is now complete:

• FID and KID measure image distributions.

• SigLIP 2 similarity measures prompt-image compatibility.

• Human reviewers assess explicitly defined prompt correctness.

This gives us three different views of the same controlled experiment.

14. The cherry on the cake: When FID preferred a model humans overwhelmingly disliked

Our synthetic example demonstrates a mathematical limitation.

But can FID actually disagree with humans when evaluating sophisticated image generators?

Yes.

A particularly striking published experiment appeared in:

Rethinking FID: Towards a Better Evaluation Metric for Image Generation, by Sadeep Jayasumana and colleagues, presented at CVPR 2024.

Although this is historical research rather than a new 2026 benchmark, the result remains a useful demonstration of why modern evaluation systems should not rely on FID alone.

14.1 The experiment

The researchers compared two configurations of Muse, an iterative text-to-image generator.

Model A: The full configuration, using more refinement iterations.

Model B: An intentionally early-stopped configuration producing less refined images.

Muse is an iterative masked-token generation system, not a conventional diffusion sampler. Nevertheless, its refinement behavior illustrates a broader concern relevant to iterative image generation.

The researchers used 1,633 prompts from PartiPrompts.

Each prompt comparison involved three human raters.

The raters evaluated image quality without being told which configuration produced each output.

14.2 The surprising results

The study reported:

The remaining 0.6% of human comparisons were indifferent.

The result is remarkable.

Human reviewers preferred the full model in 92.5% of comparisons.

Yet conventional FID assigned a lower, supposedly better, score to the intentionally degraded configuration.

The unbiased FID∞ variant made the same mistake, so the disagreement is not just a finite-sample bias of the classic estimator.

The proposed CMMD metric agreed with the direction of human preference.

14.3 Why did this happen?

The researchers identified several weaknesses in conventional FID:

• Inception features do not fully represent the complexity of modern generated content.

• Gaussian feature-distribution assumptions discard important information.

• Finite-sample behavior can affect scores and rankings.

• Changes that improve perceptual quality may move feature statistics in an unfavorable direction.

They also studied progressive distortions.

In some cases, adding distortion initially decreased FID, making degraded image sets appear statistically closer to the reference distribution.

This does not imply that FID always ranks models incorrectly.

It establishes a narrower but crucial point:

A lower FID is not sufficient evidence that humans will prefer a generator’s outputs.

14.4 What CMMD changed

The researchers proposed CLIP Maximum Mean Discrepancy, or CMMD.

CMMD compares collections of CLIP-derived image embeddings using a kernel-based distributional discrepancy.

Its differences from conventional FID include:

• A semantically richer representation.

• No requirement to approximate the feature distributions as Gaussian.

• A kernel-based comparison.

• An unbiased squared-MMD estimator under its sampling assumptions.

In the Muse experiment, CMMD ranked the models in the same direction as human preference.

However, CMMD should not be confused with CLIPScore.

CLIPScore measures prompt-image similarity.

CMMD compares two image distributions.

Using related feature encoders does not make these metrics equivalent.

The broader lesson is not to replace one unquestioned metric with another.

It is to validate automated rankings against the kinds of human judgments and task outcomes that matter.

15. Modern compositional evaluation in 2025–2026

Image generators have improved significantly at producing plausible scenes.

However, visual realism and compositional correctness remain distinct challenges.

Consider the prompt:

Exactly three ceramic mugs on a wooden table, with the blue mug between two yellow mugs.

A generator might create an attractive scene containing ceramic mugs but fail the explicit instructions.

A structured evaluator can break the prompt into separate requirements:

1. Are exactly three mugs visible?

2. Are two mugs yellow?

3. Is one mug blue?

4. Is the blue mug between the yellow mugs?

5. Are the mugs placed on a wooden table?

This is more diagnostic than one global similarity score.

15.1 T2I-CompBench++

T2I-CompBench++, published in 2025, focuses on compositional text-to-image generation.

Its 8,000 prompts cover categories including:

• Attribute binding.

• Object relationships.

• Spatial relationships.

• Numeracy.

• Complex compositions.

Its evaluation approaches include detection-based measurements and multimodal-model assessments.

The benchmark illustrates an important principle: when the task has explicit components, measure those components rather than relying solely on holistic visual similarity.

However, detection and visual question-answering systems can also make mistakes.

A failure reported by the evaluator does not always imply a failure by the image generator.

15.2 GenEval 2 and benchmark drift

A particularly relevant development is GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation, introduced in December 2025.

The researchers investigated an important problem.

As image generators improve, an established benchmark’s evaluator may become less reliable at recognizing correctness.

The benchmark can then measure the limitations of its judge rather than the capabilities of the image generator.

In their analysis of the original GenEval framework, the researchers reported absolute evaluation error reaching 17.7% for some contemporary-model comparisons.

They also found evidence that the earlier benchmark had become insufficiently challenging for newer systems.

GenEval 2 responds with:

• A new collection of 800 prompts.

• Improved coverage of objects, attributes, counts, and relationships.

• More challenging compositional requirements.

• The Soft-TIFA evaluation method.

The official GenEval 2 repository provides the benchmark materials and evaluation code.

15.3 What Soft-TIFA does differently

Soft-TIFA evaluates visual primitives and combines their judgments.

Instead of asking only a broad question such as whether an image matches a prompt, it examines whether relevant pieces of visual content are present and correct.

This makes the evaluation more interpretable.

Suppose a prompt requests:

A red suitcase below a green shelf, with two books on top.

A structured evaluation can inspect:

• Suitcase presence.

• Suitcase color.

• Shelf presence.

• Shelf color.

• Relative position.

• Number of books.

A model could pass the object-presence requirements while failing the spatial ones.

That distinction provides actionable information for model development.

15.4 Modern multimodal judges

Multimodal language models can serve as evaluators by receiving an image, the source prompt, and an explicit rubric.

They may be asked to produce judgments about:

• Object counts.

• Color assignments.

• Spatial relationships.

• Text rendering.

• Visual anomalies.

• Overall instruction adherence.

For example, a judge could return structured assessments of each required attribute rather than one overall rating.

However, multimodal judges introduce new risks.

They may hallucinate nonexistent objects, miscount repeated items, misunderstand spatial relationships, or favor particular visual styles.

Their outputs may also vary with prompt wording and model revisions.

A reliable protocol should therefore:

1. Freeze the judge model and scoring instructions.

2. Validate the judge against human-annotated examples.

3. Measure evaluator error across task categories.

4. Retain the original judge responses for auditing.

5. Reassess the judge when the generated-image distribution changes.

Automated judges are valuable tools, but they should not become unexamined sources of ground truth.

16. Additional metrics worth knowing

FID, CLIPScore, and human preference provide a useful foundation.

Modern evaluation pipelines can benefit from additional metrics when they address a specific measurement problem.

16.1 Kernel Inception Distance

KID compares distributions using a polynomial-kernel maximum mean discrepancy.

Its advantages include avoiding the Gaussian fitting step and having an unbiased estimator of squared MMD under the usual assumptions.

KID can be useful when sample-size behavior is a concern.

However, the Muse experiment demonstrates that KID is not guaranteed to agree with human preference.

16.2 CLIP-based Fréchet distances

A CLIP-based Fréchet distance replaces Inception features with image embeddings from a CLIP-family encoder.

This may provide representations that are better suited to some modern visual content.

However, the Gaussian approximation remains.

Changing the feature extractor does not eliminate all the limitations of Fréchet-based evaluation.

Furthermore, different CLIP-family checkpoints produce different feature spaces and therefore different numerical scales.

16.3 CMMD

CMMD uses kernel-based distribution comparison in a CLIP-derived feature space.

It avoids the Gaussian-fitting requirement.

The 2024 FID study found improved agreement with human judgments in several experiments.

Nevertheless, CMMD still measures distributional similarity rather than exact prompt satisfaction.

A generator can have good distributional metrics while failing individual instructions.

16.4 Learned preference predictors

Learned preference models can be useful for:

• Ranking candidate generations.

• Rapid model-development feedback.

• Prioritizing examples for manual review.

• Regression testing.

• Identifying potential visual-quality problems.

However, they may inherit aesthetic and content biases from their training data.

The ICCV 2025 study discussed earlier provides concrete evidence that some established reward-model architectures can undervalue rich visual content under particular evaluation conditions.

If such a model is used to optimize a generator directly, the generator may learn to exploit its scoring preferences.

Independent human audits remain important.

16.5 Diversity and coverage

A model can produce individually impressive images while generating a narrow range of compositions.

Useful complementary measurements include:

• Near-duplicate detection.

• Feature-space diversity.

• Coverage-oriented distribution metrics.

• Per-category output frequencies.

• Conditional diversity across repeated generations of the same prompt.

No single diversity statistic proves that all meaningful modes of an image distribution are covered.

For complex applications, inspect diversity jointly with prompt adherence and perceptual quality.

16.6 Choosing supplementary metrics

The goal is not to maximize the number of reported metrics.

Every metric should answer a relevant question.

17. Designing a trustworthy generative vision evaluation protocol

The scripts above are sufficient to experiment with the metrics.

A credible research or production evaluation requires additional controls.

17.1 Use a shared prompt distribution

Suppose Model A receives mostly landscape prompts while Model B receives complex multi-object instructions.

Their results are not directly comparable, even if both generate the same number of images.

Use a shared, versioned prompt suite.

The suite should represent the expected application.

A product-image generator might require categories for:

• Single products on clean backgrounds.

• Multiple products arranged together.

• Precise colors and materials.

• Text and branding accuracy.

• Requested camera angles.

• Spatial relationships.

• Complex commercial compositions.

A creative illustration application would need a different distribution.

The benchmark should reflect actual requirements rather than an arbitrary selection of appealing prompts.

17.2 Control generation settings

For every model, record:

• Model identifier and checkpoint revision.

• Prompt and any hidden prompt templates.

• Random seed configuration.

• Number of outputs per prompt.

• Output resolution.

• Sampling algorithm and step count.

• Guidance settings, where applicable.

• Postprocessing and upscaling operations.

• Inference precision and relevant hardware details.

For diffusion systems, scheduler settings can be particularly important.

Two outputs from the same checkpoint may differ significantly because of sampling configuration alone.

Do not select the best image from several attempts unless that selection process is part of the workflow being evaluated.

If best-of-N selection is used, include the selection mechanism and N in the benchmark specification.

17.3 Match the reference distribution

FID requires a meaningful reference distribution.

Imagine comparing generated product illustrations against natural landscape photographs.

A high FID would not necessarily imply that the illustrations are poor.

It might simply reflect a mismatch between the evaluation reference and the intended task.

Select reference images that represent the desired output domain.

Document the reference-data provenance and any filtering decisions.

17.4 Prevent evaluation contamination

If generated outputs reproduce reference images, distributional scores may be misleadingly favorable.

A low FID does not prove originality or generalization.

Relevant safeguards include:

• Held-out reference datasets.

• Exact and near-duplicate checks.

• Documented data provenance.

• Separation of development and final evaluation suites.

• Avoiding repeated tuning on the final test set.

The same concern applies to automated semantic judges.

If a generator is repeatedly optimized against one frozen evaluator, it may learn behaviors that exploit the evaluator’s weaknesses.

17.5 Report uncertainty

Suppose two models achieve FID values of 14.8 and 15.0.

That difference alone is not sufficient evidence of meaningful superiority.

The result may depend on sample count, generation randomness, and implementation details.

For robust reporting:

• Measure sensitivity to sample size.

• Repeat evaluation using independent generated samples when feasible.

• Report uncertainty or stability diagnostics.

• Keep feature extraction and preprocessing fixed.

• Avoid interpreting tiny score differences without evidence of stability.

For FID, naive bootstrapping does not automatically correct finite-sample bias.

For datasets with multiple outputs per prompt, prompt-level or hierarchical resampling may better reflect the intended uncertainty.

17.6 Analyze subgroups

An aggregate score can conceal regressions.

Imagine Model B improving dramatically on single-object photography while becoming worse at multi-object spatial relationships.

Its average semantic score might remain unchanged.

Break down results by relevant categories.

Subgroup analysis transforms benchmark measurements into engineering decisions.

17.7 Keep a human-audited benchmark subset

A modern evaluation suite should include a stable human-annotated subset.

This subset serves two purposes.

First, it provides an independent quality measurement.

Second, it helps determine whether automated evaluators remain reliable as image generators improve.

For example, if a new image model appears dramatically worse according to an automated judge but reviewers consistently prefer it, the discrepancy deserves investigation.

This is especially important in light of the benchmark drift documented by GenEval 2.

18. Reading disagreements between metrics

One of the most valuable practices in generative vision evaluation is investigating cases where different metrics disagree.

Disagreement is not necessarily a problem with the experimental results.

It may reveal that different properties of the generator have changed.

Case A: Lower FID, lower semantic similarity

Possible explanation:

The generated image distribution moved closer to the reference distribution, but the model became less faithful to individual prompts.

For example, it may produce visually conventional scenes while ignoring unusual instructions.

Recommended investigation:

• Review prompt-level outputs.

• Inspect attribute-binding failures.

• Compare human judgments of instruction fidelity.

Case B: Higher semantic similarity, lower human preference

Possible explanation:

The model includes more recognizable prompt-related content but produces poorer composition, realism, or aesthetics.

For example, every required object may be present, but the image may look cluttered or unnatural.

Recommended investigation:

• Separate human fidelity and aesthetic judgments.

• Examine cases with high semantic scores but low human ratings.

• Check whether the similarity metric rewards superficial object presence.

Case C: Lower FID, unchanged human preference

Possible explanation:

The distributional improvement may reflect features that reviewers do not notice or consider important.

Alternatively, the human study may be too small to resolve the difference.

Recommended investigation:

• Check FID stability across samples.

• Increase human comparisons.

• Inspect which visual subgroups changed.

• Avoid equating statistical improvement with demonstrated user value.

Case D: Higher human preference, worse FID

Possible explanation:

The generator became more visually attractive or more instruction-faithful while moving away from the chosen reference distribution.

The Muse experiment provides a documented example of this broad pattern.

Recommended investigation:

• Verify preference robustness.

• Inspect the reference dataset.

• Compare complementary distribution metrics.

• Report the disagreement explicitly.

A practical decision rule

Do not automatically average FID, semantic similarity, and human preference into one composite score.

They have different units, directions, assumptions, and interpretations.

A composite score requires an explicitly defined objective and justified weighting scheme.

Otherwise, the combined result may hide important failures.

A multidimensional evaluation report is often more useful than a single leaderboard number.

19. From experimental scripts to a production evaluation system

Once an organization regularly develops generative image systems, evaluation should become a repeatable engineering process.

A practical architecture has five stages.

Stage 1: Dataset and prompt versioning

Maintain a versioned manifest containing:

• Prompt text.

• Prompt category.

• Evaluation criteria.

• Random seed schedule.

• Output-generation configuration.

Changing the benchmark should create a new benchmark version rather than silently modifying historical results.

Stage 2: Automated generation

Run candidate models against the shared prompt suite.

Store outputs with stable identifiers.

Record checkpoint revisions, sampling configurations, resolutions, and compute settings.

For diffusion systems, include scheduler and guidance configuration.

Stage 3: Automated scoring

Calculate relevant metrics.

A modern scoring pipeline might include:

• FID or another distribution-level measure.

• KID or CMMD.

• A documented vision-language similarity score.

• Compositional correctness.

• Diversity measurements.

• Generation latency and resource usage.

Store the results with the exact scoring configuration.

Stage 4: Human audit

Select comparisons for blinded human evaluation.

Useful sampling strategies include:

• Uniform random prompts.

• Prompts with large differences between models.

• Cases where automated metrics disagree.

• Underrepresented task categories.

• Known failure modes.

Be careful with enriched samples.

A study deliberately focused on difficult examples should not be treated as an unbiased estimate of general user preference unless sampling weights or a separate representative study justify that interpretation.

Stage 5: Release decisions

Define acceptance criteria before examining final test results.

For example, a new model might be required to:

• Avoid a meaningful regression in primary human preference.

• Meet minimum compositional accuracy requirements.

• Stay within latency and cost limits.

• Avoid new systematic rendering failures.

• Preserve acceptable diversity across prompt categories.

Predefined requirements reduce the temptation to select whichever metric makes the preferred model look best.

20. Common implementation mistakes

Even a conceptually appropriate metric can become misleading through implementation errors.

Mistake 1: Comparing FID scores from different pipelines

Different preprocessing or feature-extraction implementations can produce incomparable results.

Fix: Freeze and document the complete evaluation configuration.

Mistake 2: Using conventional FID on tiny datasets

A small sample combined with 2,048-dimensional features produces rank-deficient covariance estimates.

Fix: Use an appropriate small-sample metric such as KID, and distinguish diagnostic reduced-feature FID from conventional benchmark FID.

Mistake 3: Treating CLIPScore as a probability

An embedding similarity is not a calibrated probability of instruction correctness.

Fix: Interpret it as a relative signal and validate it against the target task.

Mistake 4: Mixing encoder backbones

CLIPScore computed with one checkpoint is not directly comparable to similarity computed with another.

Fix: Record the exact encoder, checkpoint revision, preprocessing, and scoring transformation.

Mistake 5: Treating every reviewer vote as independent

Multiple judgments of the same prompt share common sources of variation.

Fix: Preserve prompt and reviewer identifiers and use an appropriate uncertainty method.

Mistake 6: Reusing one fixed left/right assignment

Showing the same image order to all reviewers can introduce systematic presentation bias.

Fix: Counterbalance or randomize image positions across reviewers.

Mistake 7: Ignoring diffusion inference settings

Different step counts, guidance settings, or schedulers can change image quality significantly.

Fix: Record generation configuration alongside model identity.

Mistake 8: Trusting a multimodal judge without validation

Automated judges can miscount objects, misunderstand relationships, or hallucinate details.

Fix: Validate them on human-annotated examples and monitor benchmark drift.

Mistake 9: Reporting only favorable metrics

Choosing whichever measurement makes a candidate look better is not rigorous evaluation.

Fix: Predefine primary and secondary measurements and report disagreements.

Mistake 10: Publishing an unpinned environment

A script that works today may behave differently after dependency updates.

Fix: Pin package versions, lock transitive dependencies for the target platform, record model revisions, and run automated smoke tests.

21. A practical evaluation checklist

Before publishing a benchmark comparison or deploying a new image generator, verify the following.

Evaluation readiness

• Both models were evaluated on the same prompt suite.
• Model checkpoints and inference configurations were recorded.
• Diffusion sampling steps, guidance settings, and seeds were documented.
• Reference images represent the intended output domain.
• FID feature dimensions and preprocessing are identical.
• Small-sample numerical limitations were considered.
• Semantic similarity uses the same encoder and scoring definition for both models.
• Human judgments are blinded and presentation positions are counterbalanced.
• Reviewer and prompt identifiers are preserved.
• Uncertainty intervals were calculated using an appropriate sampling structure.
• Important prompt categories were evaluated separately.
• Automated judge performance was audited against human judgments.
• Metric disagreements were investigated.
• Package versions, checkpoint revisions, and benchmark versions were recorded.
• Final release criteria were specified before inspecting the results.

This checklist is not a substitute for statistical rigor, but it provides a practical review framework for evaluating generative vision systems.

22. Final recommendations

FID, CLIPScore, and human preference each provide valuable information, but none fully captures generative image quality.

FID remains useful for distribution-level evaluation when reference images, feature extraction, preprocessing, and sample counts are carefully controlled.

However, conventional FID has statistical limitations, especially with small samples and high-dimensional feature representations.

CLIPScore and newer vision-language similarity methods provide prompt-conditioned signals.

They can identify semantic mismatches that distributional metrics cannot detect.

But a high similarity score is not proof that every object, count, attribute, and spatial relationship is correct.

Using a newer encoder such as SigLIP 2 can modernize the representation, provided that the resulting score is documented as a distinct metric rather than presented as interchangeable with historical CLIPScore.

Human preference measures judgments relevant to a defined user task.

It is often the most consequential evaluation signal for product decisions, provided the study is blinded, representative, and statistically sound.

Modern compositional benchmarks and multimodal judges add another important dimension.

They help evaluate whether images actually satisfy detailed instructions.

The development of GenEval 2 and Soft-TIFA illustrates why evaluation benchmarks must evolve alongside image-generation models.

For diffusion-based generation, quality measurements must also account for the sampling procedure.

A model checkpoint alone does not determine the final output distribution.

Sampling steps, guidance settings, schedulers, random seeds, and postprocessing can all affect observed performance.

The strongest evaluation systems do three things consistently:

1. Measure distinct properties instead of collapsing everything into one vaguely defined quality score.

2. Use reproducible protocols and report statistical uncertainty.

3. Treat disagreements between automated metrics and human judgments as valuable diagnostic evidence.

Our controlled experiment makes the first principle concrete.

Two systems can contain precisely the same images while one associates those images with the wrong prompts.

Distributional metrics cannot detect that mapping error because the marginal image distributions are identical.

The published Muse experiment makes the lesson even more striking.

A widely used metric preferred an intentionally degraded generator, while human reviewers overwhelmingly preferred the full model.

The objective of evaluation is not to produce the most impressive benchmark number.

It is to measure whether a generative system is becoming better at the tasks that matter.

23. Further reading and references

The following resources provide implementation details, research evidence, and modern evaluation approaches.

Current implementation resources

TorchMetrics: Official repository

Maintained PyTorch metric implementations, including FID and KID. The examples in this tutorial use the APIs associated with the 1.9.0 release.

Torch-fidelity 0.4.0

The February 2026 release of the feature-extraction and generative evaluation library used with TorchMetrics.

Transformers 5.x: SigLIP 2 documentation

The documented modern vision-language inference interface used in our Python implementation.

SigLIP 2: Multilingual Vision-Language Encoders

The 2025 research introducing the model family used for modern text-image similarity scoring.

Evaluation research

Rethinking FID: Towards a Better Evaluation Metric for Image Generation

Jayasumana and colleagues, CVPR 2024. Historical but foundational evidence that conventional FID can disagree strongly with human preference.

T2I-CompBench++

A 2025 benchmark focused on compositional text-to-image evaluation, including attributes, relationships, and numeracy.

GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluation

Research introduced in December 2025 investigating automated benchmark drift and introducing more challenging compositional evaluation with Soft-TIFA.

GenEval 2: Official repository

Benchmark prompts, evaluation code, and practical implementation materials.

Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignment

Ba and colleagues, ICCV 2025. Research examining limitations of preference reward models when evaluating visually rich, high-quality generated images.

24. Your next step: Build an evaluation report that explains quality, not just scores

Start by creating the pinned Python environment and running the synthetic benchmark.

python verify_env.py
python make_demo_data.py
python evaluate.py
python prepare_human.py

Inspect the anonymous image comparisons, collect reviewer responses, and calculate human preference.

python summarize_human.py

Once the pipeline works, replace the synthetic shapes with outputs from two real generative image models.

For diffusion models, use a predefined prompt suite and record sampling steps, guidance configuration, scheduler settings, seeds, and output resolution.

Generate multiple representative samples per prompt when resources allow.

Calculate distributional metrics, semantic similarity, compositional correctness, and blinded human preference.

Your first complete evaluation report should contain:

• A distribution-level measurement with its precise feature and preprocessing configuration.

• A semantic alignment measurement with the encoder and checkpoint revision recorded.

• Per-category compositional correctness results.

• Human preference statistics with uncertainty intervals.

• A gallery of representative failures.

• Generation latency and compute information where relevant.

• An investigation of cases where automated scores and human judgments disagree.

Your challenge: Find at least one prompt for which the model with the better automated score generates the image that human reviewers prefer less.

Determine whether the disagreement comes from visual quality, prompt adherence, object relationships, reference-distribution mismatch, or an automated evaluator’s limitations.

Then document the failure and add a regression test.

That investigation will teach you more about evaluating generative vision systems than optimizing any single score ever could.