Benchmarking Single-Factor Physical Video-to-Audio Generation

• URL: http://arxiv.org/abs/2605.30339v1
• Date: Thu, 28 May 2026 17:59:09 GMT
• Title: Benchmarking Single-Factor Physical Video-to-Audio Generation
• Abstract summary: Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes.<n>We introduce a benchmark that audits the physical reasoning of V2A models through controlled counterfactual pairs.<n>Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels.
• Score: 40.20239686648815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. In this paper, we introduce FlatSounds, a benchmark that audits the physical reasoning of V2A models through: 1) controlled counterfactual pairs in which a single physical factor is varied, and 2) single-video pattern tests that probe internal consistency and directional trends. These settings test whether the generated audio correctly reflects specific physical properties and timings. Our evaluation of state-of-the-art models reveals a consistent trade-off: models rely more on text captions than the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment. Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels. Finally, we find that our physics-based metrics correlate strongly with human preference tests on our own data. Project webpage: https://research.nvidia.com/labs/cosmos-lab/flatsounds/

Related papers

• Do Joint Audio-Video Generation Models Understand Physics? [49.44016960193229]
  Joint audio-video generation models are rapidly approaching professional production quality.<n>Do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency?<n>We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation.
  arXiv  Detail & Related papers  (2026-05-08T00:14:07Z)
• MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimation [35.2836152950695]
  Video-to-audio (V2A) models do not explicitly model room-acoustic effects such as reverberation or room impulse responses (RIRs)<n>In this paper, we propose the way to utilize the pretrained model as prior for physically grounded room-acoustic processing.
  arXiv  Detail & Related papers  (2026-05-01T06:06:56Z)
• Physical Simulator In-the-Loop Video Generation [96.87054314612142]
  Physical Simulator In-the-loop Video Generation (PSIVG) is a novel framework that integrates a physical simulator into the video diffusion process.<n>PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity.
  arXiv  Detail & Related papers  (2026-03-06T15:48:25Z)
• PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation [63.3417467957431]
  Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content.<n>We present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to evaluate the audio physics grounding capabilities of existing T2AV models.<n>Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation.
  arXiv  Detail & Related papers  (2025-12-30T05:22:31Z)
• PAVAS: Physics-Aware Video-to-Audio Synthesis [58.746986798623084]
  We present Physics-Aware Video-to-Audio Synthesis (PAVAS), a method that incorporates physical reasoning into a latent diffusion-based V2A generation.<n>We show that PAVAS produces physically plausible and perceptually coherent audio, outperforming existing V2A models in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-12-09T06:28:50Z)
• LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference [57.086932851733145]
  We introduce LikePhys, a training-free method that evaluates intuitive physics in video diffusion models.<n>We benchmark intuitive physics understanding in current video diffusion models.<n> Empirical results show that, despite current models struggling with complex and chaotic dynamics, there is a clear trend of improvement in physics understanding as model capacity and inference settings scale.
  arXiv  Detail & Related papers  (2025-10-13T15:19:07Z)
• Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representations [16.447597767676655]
  We aim to pinpoint the source location in the input video sequence. Recent works have shown impressive audio-visual separation results when using prior knowledge of the source type. We propose a two-stage architecture, called Appearance and Motion network (AMnet), where the stages specialise to appearance and motion cues.
  arXiv  Detail & Related papers  (2021-04-17T10:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.