Inference-Time Scaling for Joint Audio-Video Generation

• URL: http://arxiv.org/abs/2606.03183v1
• Date: Tue, 02 Jun 2026 05:41:41 GMT
• Title: Inference-Time Scaling for Joint Audio-Video Generation
• Abstract summary: Joint audio-video generation models often require substantial training resources to improve fidelity.<n>Inference-Time Scaling is a promising training-free alternative in single-modality domains.<n>We present the first comprehensive study of ITS for joint audio-video generation.
• Score: 38.09471807128537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Joint audio-video generation aims to synthesize realistic audio-video pairs that are both semantically aligned with text prompts and precisely synchronized. While existing joint audio-video generation models often require substantial training resources to improve fidelity, Inference-Time Scaling (ITS) has recently emerged as a promising training-free alternative in single-modality domains. However, extending ITS from a single modality to multimodal domains is non-trivial, as it requires balancing multiple heterogeneous objectives. In this paper, we present the first comprehensive study of ITS for joint audio-video generation. We first demonstrate that a multi-verifier framework is essential to address the limitations of single-objective guidance, including asymmetric performance trade-offs and verifier hacking. Through systematic analysis, we then identify an optimal multi-verifier combination that yields balanced improvements across all quality dimensions. Finally, to effectively aggregate diverse reward signals, we propose Adaptive Reward Weighting (ARW), a novel test-time optimization algorithm. ARW treats reward aggregation as an online optimization problem, utilizing learnable parameters to calibrate reward variances without requiring prior knowledge of reward distributions, thereby ensuring robust multi-objective selection. Experimental results on VGGSound and JavisBench-mini benchmarks demonstrate that our framework significantly enhances semantic alignment, perceptual quality, and audio-visual synchronization of generated outputs. Synthesized samples and code are available on the project page: https://jung-jaemin.github.io/ITS-AVGen-Proj.

Related papers

• NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation [10.472880594575196]
  Diffusion models have recently advanced text-to-video (T2V) generation, yet they still struggle with fine-grained compositional alignment.<n>We propose NoisEasier, a test-time scaling framework that improves T2V generation through differentiable reward-guided noise optimization.
  arXiv  Detail & Related papers  (2026-08-31T03:17:52Z)
• VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation [79.34043102155964]
  We propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation.<n>We show VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations.
  arXiv  Detail & Related papers  (2026-08-19T06:53:14Z)
• MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning [15.707226798418]
  Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning.<n>Existing methods rely on modality-specific encoders and complex combinations of contrastive or reconstruction objectives.<n>We introduce MJEPA, a joint-embedding predictive architecture for audio-visual learning that uses a single, unified encoder for both modalities.
  arXiv  Detail & Related papers  (2026-06-23T22:48:42Z)
• Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation [57.61006679699323]
  Reinforcement learning has emerged as an effective framework for post-training visual generative models with human preference signals.<n>In modern visual generation, multiple reward models are often used to capture heterogeneous objectives, such as visual quality, motion consistency, and text alignment.<n>We propose Objective-aware Trajectory Credit Assignment (OTCA), a structured framework for fine-grained GRPO training.
  arXiv  Detail & Related papers  (2026-04-21T08:37:13Z)
• JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation [112.614973927778]
  Joint audio-video generation (JAVG) produces synchronized and semantically aligned sound and vision from textual descriptions.<n>This paper presents JavisDiT++, a framework for unified modeling and optimization of JAVG.<n>Our model achieves state-of-the-art performance merely with around 1M public training entries.
  arXiv  Detail & Related papers  (2026-02-22T12:44:28Z)
• GMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining [64.72014392166625]
  GMS-CAVP is a novel framework that combines Multi-Scale Video-Audio Alignment and Multi-Scale Spatial-Temporal Diffusion-based pretraining objectives.<n>First, GMS-CAVP introduces a multi-scale contrastive learning strategy that captures semantic and temporal relations across varying granularities.<n>Second, we go beyond traditional contrastive learning by incorporating a diffusion-based generative objective, enabling modality translation and synthesis between video and audio.
  arXiv  Detail & Related papers  (2026-01-27T13:43:32Z)
• PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation [57.864929968616586]
  Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions.<n>We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning.
  arXiv  Detail & Related papers  (2025-11-24T07:11:12Z)
• Tailored Design of Audio-Visual Speech Recognition Models using Branchformers [0.0]
  We propose a novel framework for the design of parameter-efficient Audio-Visual Speech Recognition systems.<n>To be more precise, the proposed framework consists of two steps: first, estimating audio- and video-only systems, and then designing a tailored audio-visual unified encoder.<n>Our models achieve competitive word error rates (WER) of approximately 2.5% for English and surpass existing approaches for Spanish.
  arXiv  Detail & Related papers  (2024-07-09T07:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.