CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

• URL: http://arxiv.org/abs/2608.16829v1
• Date: Mon, 17 Aug 2026 17:14:50 GMT
• Title: CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?
• Abstract summary: CaliBench scores outcomes in a discrete space rather than a learned feature space such as in FID.<n>We decompose performance into two axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample.<n>We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each.
• Score: 6.39120343569684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.

Related papers

• Cross-Block Conditioning in Deep Boltzmann Machines for Statistical Data Fusion [0.0]
  We propose observed-block multi-prediction, which restricts the multi-prediction objective to targets drawn from what each row actually observes.<n>Having a discriminative criterion that survives the setting lets us ask whether the joint model is needed at all.
  arXiv  Detail & Related papers  (2026-09-14T02:26:16Z)
• PAWBench: How Far Are We from Probabilistically Aligned World Modeling? [60.16773899025656]
  We call this distribution-level requirement probabilistic alignment.<n>PAWBench is a benchmark for evaluating video generators as samplers of world dynamics.<n>We introduce PAWEval, an outcome-level protocol that converts repeated video rollouts into empirical distributions.
  arXiv  Detail & Related papers  (2026-08-27T16:46:38Z)
• EDGE: a closed-form directed test for the calibration of probabilistic binary classifiers [0.0]
  We propose EDGE, a calibration test for the canonical probabilistic classifier, logistic regression.<n> EDGE reads the same binned predicted-observed table a reliability diagram plots, and projects its standardized bin residuals onto a small pre-specified basis of smooth calibration-distortion.<n>Across link and feature misspecification the pre-specified default led or tied every rival binned test on the fitted index in 19 of 22 detectable scenarios, and stayed computable where the refit-based Stukel score test separates in 20% to 28% of samples.
  arXiv  Detail & Related papers  (2026-08-20T19:06:05Z)
• The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping [70.07961857926377]
  We introduce trace-grounded parametric profiling for event counting in three controlled video tasks.<n>Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed.<n>At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events.
  arXiv  Detail & Related papers  (2026-08-06T17:57:06Z)
• Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting [0.0]
  Training data for machine learning is routinely collected by a selection process the model never sees.<n>Etonians solved the harder case in 1979: Heckman's two-equation model fits a probit selection equation and an outcome equation linked through correlated errors.<n>The Heckman correction restores coverage when the selection equation has an instrument -- a variable affecting selection but not the outcome -- and degrades measurably without one.
  arXiv  Detail & Related papers  (2026-07-07T03:59:46Z)
• When the Next Step Is Not One Step: Distribution-Aware Execution Modeling for Concurrent Go Programs [0.0]
  Training a model to predict the next step in a concurrent program is harder than it looks, because the scheduler is nondeterministic.<n>We run each program many times, aggregate the observed next events into an empirical distribution, and fine-tune a 7B model to match that distribution with a KL objective.<n>On 798 held-out predictions drawn from real production Go bugs, fine-tuning on fewer than a thousand traces reaches 36.2% accuracy.<n>We derive a formal goroutine-leak signature for a class of select-blocked goroutines where P(GoUnblock) holds by scheduler semantics, not by learning
  arXiv  Detail & Related papers  (2026-06-16T04:40:04Z)
• DTVEM-RE: A Hierarchical Random-Effects Extension of the Differential Time-Varying Effect Model for Person-Specific Multi-Lag Estimation in Intensive Longitudinal Data [0.0]
  We present DTVEM-RE, an extension that lets each person have their own lag coefficients.<n>It is the first person-specific implementation of DTVEM-style lag detection.
  arXiv  Detail & Related papers  (2026-06-12T04:54:21Z)
• Resolving Predictive Multiplicity for the Rashomon Set [25.615527704500987]
  A Rashomon set'' of models achieve similar accuracy but diverges in their individual predictions.<n>We propose three approaches to reduce inconsistency among predictions for the members of the Rashomon set.
  arXiv  Detail & Related papers  (2026-01-14T02:02:54Z)
• It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models [51.66015254740692]
  We show that for an ensemble of deep learning based classification models, bias and variance are emphaligned at a sample level. We study this phenomenon from two theoretical perspectives: calibration and neural collapse.
  arXiv  Detail & Related papers  (2023-10-13T17:06:34Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)
• Proximity-Informed Calibration for Deep Neural Networks [49.330703634912915]
  ProCal is a plug-and-play algorithm with a theoretical guarantee to adjust sample confidence based on proximity. We show that ProCal is effective in addressing proximity bias and improving calibration on balanced, long-tail, and distribution-shift settings.
  arXiv  Detail & Related papers  (2023-06-07T16:40:51Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.