The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

• URL: http://arxiv.org/abs/2606.20536v1
• Date: Thu, 18 Jun 2026 17:49:32 GMT
• Title: The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
• Abstract summary: We treat FID as a random variable on a two-axis panel of training and generation seeds.<n>We measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256.
• Score: 55.62875959637507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.

Related papers

• What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation [6.078726431806193]
  Generative models are commonly ranked by Fréchet Inception Distance (FID) and Kernel Inception Distance (KID)<n>ZID detects a broad range of departures, and its score tracks increasing severity along the corresponding sweeps.
  arXiv  Detail & Related papers  (2026-08-25T17:58:46Z)
• Tokenizer Generator Coupling in Medical Image Generation [0.0]
  Latent medical image generators usually treat the tokenizer as fixed preprocessing.<n>We test whether this separation is valid in a controlled ChestMNIST study at 64x64.<n> rankings depend jointly on the tokenizer, generator, and sampler.
  arXiv  Detail & Related papers  (2026-08-07T18:57:09Z)
• Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration [88.94434023253872]
  We propose Group Diffusion, unlocking the attention mechanism to be shared across images.<n>We observe a clear scaling effect - larger group sizes yield stronger cross-sample attention and better generation quality.<n>Our work reveals cross-sample inference as an effective, previously unexplored mechanism for generative modeling.
  arXiv  Detail & Related papers  (2025-12-11T18:59:55Z)
• Measuring training variability from stochastic optimization using robust nonparametric testing [5.519968037738177]
  We propose a robust hypothesis testing framework and a novel summary statistic, the $alpha$-trimming level, to measure model similarity.<n>Applying hypothesis testing directly with the $alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis.<n>We show how to use the $alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics.
  arXiv  Detail & Related papers  (2024-06-12T15:08:15Z)
• Rethinking FID: Towards a Better Evaluation Metric for Image Generation [43.66036053597747]
  Inception Distance estimates the distance between a distribution of Inception-v3 features of real images, and those of images generated by the algorithm. We highlight important drawbacks of FID: Inception's poor representation of the rich and varied content generated by modern text-to-image models, incorrect normality assumptions, and poor sample complexity. We propose an alternative new metric, CMMD, based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel.
  arXiv  Detail & Related papers  (2023-11-30T19:11:01Z)
• Memorization Through the Lens of Curvature of Loss Function Around Samples [10.028765645749338]
  We propose using the curvature of loss function around each training sample, averaged over training epochs, as a measure of memorization of the sample. We first show that the high curvature samples visually correspond to long-tailed, mislabeled, or conflicting samples, those that are most likely to be memorized. This analysis helps us find, to the best of our knowledge, a novel failure mode on the CIFAR100 and ImageNet datasets.
  arXiv  Detail & Related papers  (2023-07-11T22:53:09Z)
• Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness [86.61582747039053]
  Language model training in distributed settings is limited by the communication cost of exchanges. We extend recent work using shared randomness to perform distributed fine-tuning with low bandwidth.
  arXiv  Detail & Related papers  (2023-06-16T17:59:51Z)
• Generalized Differentiable RANSAC [95.95627475224231]
  $nabla$-RANSAC is a differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. $nabla$-RANSAC is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives.
  arXiv  Detail & Related papers  (2022-12-26T15:13:13Z)
• Reducing Training Sample Memorization in GANs by Training with Memorization Rejection [80.0916819303573]
  We propose rejection memorization, a training scheme that rejects generated samples that are near-duplicates of training samples during training. Our scheme is simple, generic and can be directly applied to any GAN architecture.
  arXiv  Detail & Related papers  (2022-10-21T20:17:50Z)
• Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization [50.725191156128645]
  Multi-armed bandit algorithms like Thompson Sampling can be used to conduct adaptive experiments. We present simulations for 2-arm experiments that explore two algorithms that combine the benefits of uniform randomization for statistical analysis.
  arXiv  Detail & Related papers  (2021-12-15T22:11:58Z)
• Uncertainty Inspired RGB-D Saliency Detection [70.50583438784571]
  We propose the first framework to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Inspired by the saliency data labeling process, we propose a generative architecture to achieve probabilistic RGB-D saliency detection. Results on six challenging RGB-D benchmark datasets show our approach's superior performance in learning the distribution of saliency maps.
  arXiv  Detail & Related papers  (2020-09-07T13:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.