Daily Paper Briefing

Abstract Overview

This paper studies Fréchet Inception Distance (FID) as a random variable rather than a fixed score, using a two-axis panel over training seeds and sampling seeds. Across several hundred SiT diffusion models trained on class-conditional ImageNet 256x256, the authors directly measure how much FID changes when a model is retrained versus when samples are redrawn from a fixed model. They find that variability from retraining is substantially larger than variability from resampling, and they further decompose the training-side spread into initialization, data order, and flow-matching noise. The study also tests whether guidance tuning, more compute, larger models, or learning-rate transfer reduce this instability, and uses the findings to motivate a multi-seed FID reporting protocol.

Novelty

The paper’s main novelty is to operationalize FID evaluation as a two-dimensional randomness problem over both training and generation seeds, and to measure that variance directly at scale. It also introduces a per-cell golden-section guidance-tuning protocol (GS-FID) to study how guidance optimization changes the noise floor and seed rankings.

Results

On the reported SiT-B/2 panel, retraining changes FID 3.2x more than resampling from a fixed model, with an empirical training-seed CoV around 1.3% versus about 0.4% from sampling seeds alone. Flow-matching noise is the largest identified training-time source, ahead of initialization and data order, while numerical nondeterminism contributes little to FID variance. Per-cell CFG tuning roughly halves the relative spread, but rankings reshuffle, and the residual 1-2% CoV floor persists across model sizes and training budgets; a lucky seed can also reach the same FID with up to 2x less compute than an unlucky one.

Key Points

1. Training-seed variability dominates sampling-seed variability in FID evaluation, so single-model reporting understates uncertainty.
2. Among training-time randomness sources, flow-matching noise contributes the most to FID spread, followed by initialization and then data order, while hardware nondeterminism has little practical effect on FID.
3. Guidance tuning with GS-FID reduces the relative FID noise floor but changes seed rankings, and the remaining 1-2% coefficient-of-variation floor persists across scale and compute.

References

• arXiv: https://arxiv.org/abs/2606.20536v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.20536v1
• Hugging Face Papers: https://huggingface.co/papers/2606.20536
• Project: https://kyutai.org/fid-lottery