MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimation

• URL: http://arxiv.org/abs/2605.00431v1
• Date: Fri, 01 May 2026 06:06:56 GMT
• Title: MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimation
• Abstract summary: Video-to-audio (V2A) models do not explicitly model room-acoustic effects such as reverberation or room impulse responses (RIRs)<n>In this paper, we propose the way to utilize the pretrained model as prior for physically grounded room-acoustic processing.
• Score: 35.2836152950695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although recent video-to-audio (V2A) models excelled at synthesizing semantically plausible sounds from visual inputs, they do not explicitly model room-acoustic effects such as reverberation or room impulse responses (RIRs), and thus offer limited controllability over these effects. However, we hypothesize that such V2A models implicitly have semantic knowledge of the relationship between spatial audio and the corresponding vision cues. In this paper, we revisit a V2A model for the sake of the above, and propose the way to utilize the pretrained model as prior for physically grounded room-acoustic processing. Based on one of the state-of-the-art V2A models, MMAudio, we propose MMAudioReverbs that is a unified framework dealing with i) dereverberation and ii) room impulse response (RIR) estimation without network architectural modification, and fine-tuned on a small dataset. Experimental results showed that audio and visual cues respectively have advantage depending on the type of physical room acoustics. It implies that foundation V2A models can be used for physically grounded room-acoustic analysis.

Related papers

• Spatial Audio Question Answering and Reasoning on Dynamic Source Movements [8.029049649310211]
  We introduce a movement-centric spatial audio augmentation framework that synthesizes diverse motion patterns from isolated mono audio events.<n>Second, we propose an end-to-end multimodal finetuning approach with a thinking mode, which allows audio-language models to produce explicit intermediate reasoning steps.<n>Third, we investigate the impact of query-conditioned source separation as a preprocessing stage and compare three inference regimes.
  arXiv  Detail & Related papers  (2026-02-18T10:16:30Z)
• PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation [63.3417467957431]
  Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content.<n>We present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to evaluate the audio physics grounding capabilities of existing T2AV models.<n>Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation.
  arXiv  Detail & Related papers  (2025-12-30T05:22:31Z)
• PAVAS: Physics-Aware Video-to-Audio Synthesis [58.746986798623084]
  We present Physics-Aware Video-to-Audio Synthesis (PAVAS), a method that incorporates physical reasoning into a latent diffusion-based V2A generation.<n>We show that PAVAS produces physically plausible and perceptually coherent audio, outperforming existing V2A models in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-12-09T06:28:50Z)
• Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces [3.5570874721859016]
  Information content (IC) of predictions from a Generative Infinite-Vocabulary Transformer (GIVT) has been used to model musical expectancy and surprisal in audio.<n>We investigate the effectiveness of such modelling using IC calculated with autoregressive diffusion models (ADMs)<n>We evaluate diffusion model IC's effectiveness in capturing surprisal aspects by examining two tasks: (1) capturing monophonic pitch surprisal, and (2) detecting segment boundaries in multi-track audio.
  arXiv  Detail & Related papers  (2025-08-07T12:05:27Z)
• Generating Moving 3D Soundscapes with Latent Diffusion Models [4.792255437561157]
  We introduce SonicMotion, a framework capable of generating Ambisonics audio with explicit control over moving sound sources.<n>SonicMotion achieves state-of-the-art semantic alignment and perceptual quality comparable to leading text-to-audio systems.
  arXiv  Detail & Related papers  (2025-07-09T22:31:06Z)
• Differentiable Room Acoustic Rendering with Multi-View Vision Priors [12.30408352143278]
  We introduce Audio-Visual Differentiable Room Acoustic Rendering (AV-DAR), a framework that leverages visual cues extracted from multi-view images and acoustic beam tracing for physics-based room acoustic rendering.<n>Experiments across six real-world environments from two datasets demonstrate that our multimodal, physics-based approach is efficient, interpretable, and accurate.
  arXiv  Detail & Related papers  (2025-04-30T17:55:29Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene.<n>Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio.<n>We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment.<n>Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis [129.86743102915986]
  We formulate the synthesis process from a different perspective by decomposing the audio into a common part. We propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Experiment results show that BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics.
  arXiv  Detail & Related papers  (2022-05-30T02:09:26Z)
• Deliberation Model Based Two-Pass End-to-End Speech Recognition [52.45841282906516]
  A two-pass model has been proposed to rescore streamed hypotheses using the non-streaming Listen, Attend and Spell (LAS) model. The model attends to acoustics to rescore hypotheses, as opposed to a class of neural correction models that use only first-pass text hypotheses. A bidirectional encoder is used to extract context information from first-pass hypotheses.
  arXiv  Detail & Related papers  (2020-03-17T22:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.