Do Joint Audio-Video Generation Models Understand Physics?

• URL: http://arxiv.org/abs/2605.07061v1
• Date: Fri, 08 May 2026 00:14:07 GMT
• Title: Do Joint Audio-Video Generation Models Understand Physics?
• Abstract summary: Joint audio-video generation models are rapidly approaching professional production quality.<n>Do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency?<n>We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation.
• Score: 49.44016960193229
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency? We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench tests models across three scene categories: Steady State, Event Transition, and Environment Transition. It covers physics-grounded subcategories drawn from real-world scenes, plus Anti-AV-Physics prompts that deliberately request physically inconsistent audio-video behavior. Each generation is evaluated along five dimensions: visual semantic adherence, audio semantic adherence, visual physical commonsense, audio physical commonsense, and cross-modal physical commonsense. Across three proprietary and four open-source models, we find that Seedance 2.0 performs best overall, but all models remain far from robust physical understanding. Performance drops sharply on event-driven and environment-driven transitions, and even strong proprietary systems collapse on Anti-AV-Physics prompts. We further introduce AV-Phys Agent, a ReAct-style evaluator that combines a multimodal language model with deterministic acoustic measurement tools, producing rankings that closely align with human ratings. Our results identify cross-modal physical consistency and transition-driven scene dynamics as key open challenges for joint audio-video generation.

Related papers

• Benchmarking Single-Factor Physical Video-to-Audio Generation [40.20239686648815]
  Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes.<n>We introduce a benchmark that audits the physical reasoning of V2A models through controlled counterfactual pairs.<n>Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels.
  arXiv  Detail & Related papers  (2026-05-28T17:59:09Z)
• PhyCo: Learning Controllable Physical Priors for Generative Motion [55.59209981836171]
  We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation.<n>Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback.
  arXiv  Detail & Related papers  (2026-04-30T17:53:03Z)
• Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine [50.62040226184694]
  We present OmniFysics, a compact omni-modal model that unifies understanding across images, audio, video, and text.<n>To inject explicit physical knowledge, we build a physical data engine with two components.<n>Experiments show competitive performance on standard multimodal benchmarks and improved results on physics-oriented evaluations.
  arXiv  Detail & Related papers  (2026-02-05T14:04:51Z)
• PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation [63.3417467957431]
  Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content.<n>We present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to evaluate the audio physics grounding capabilities of existing T2AV models.<n>Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation.
  arXiv  Detail & Related papers  (2025-12-30T05:22:31Z)
• VABench: A Comprehensive Benchmark for Audio-Video Generation [22.00633729850902]
  VABench is a benchmark framework designed to evaluate the capabilities of synchronous audio-video generation.<n>It covers three task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation.<n>VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds.
  arXiv  Detail & Related papers  (2025-12-10T03:57:29Z)
• PAVAS: Physics-Aware Video-to-Audio Synthesis [58.746986798623084]
  We present Physics-Aware Video-to-Audio Synthesis (PAVAS), a method that incorporates physical reasoning into a latent diffusion-based V2A generation.<n>We show that PAVAS produces physically plausible and perceptually coherent audio, outperforming existing V2A models in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-12-09T06:28:50Z)
• PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selection [10.498184571108995]
  We propose PhysCorr, a unified framework for modeling, evaluating, and optimizing physical consistency in video generation.<n>Specifically, we introduce PhysicsRM, the first dual-dimensional reward model that quantifies both intra-object stability and inter-object interactions.<n>Our approach is model-agnostic and scalable, enabling seamless integration into a wide range of video diffusion and transformer-based backbones.
  arXiv  Detail & Related papers  (2025-11-06T02:40:57Z)
• Can Your Model Separate Yolks with a Water Bottle? Benchmarking Physical Commonsense Understanding in Video Generation Models [14.187604603759784]
  We present PhysVidBench, a benchmark designed to evaluate the physical reasoning capabilities of text-to-video systems.<n>For each prompt, we generate videos using diverse state-of-the-art models and adopt a three-stage evaluation pipeline.<n> PhysVidBench provides a structured, interpretable framework for assessing physical commonsense in generative video models.
  arXiv  Detail & Related papers  (2025-07-21T17:30:46Z)
• Think Before You Diffuse: Infusing Physical Rules into Video Diffusion [55.046699347579455]
  The complexity of real-world motions, interactions, and dynamics introduce great difficulties when learning physics from data.<n>We propose DiffPhy, a generic framework that enables physically-correct and photo-realistic video generation by fine-tuning a pre-trained video diffusion model.
  arXiv  Detail & Related papers  (2025-05-27T18:26:43Z)
• VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior [88.51778468222766]
  Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos.<n>VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics.<n>We propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior.
  arXiv  Detail & Related papers  (2025-03-30T09:03:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.