BinauralVAE: Spatial Audio Reconstruction For World Models

• URL: http://arxiv.org/abs/2609.06837v1
• Date: Sun, 06 Sep 2026 21:20:36 GMT
• Title: BinauralVAE: Spatial Audio Reconstruction For World Models
• Abstract summary: We introduce BinauralVAE: a flexible, open-source pipeline that explores multiple models for spatialized audio reconstruction.<n>Our approach evaluates various Variational Autoencoder architectures to learn robust representations of latent signals.<n>This pipeline establishes a foundation for state representation in a future audio-based world model.
• Score: 2.725160922570703
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

Related papers

• Audio-Visual Intelligence in Large Foundation Models [123.78167820053545]
  Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence.<n>Despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent, and heterogeneous evaluation practices.<n>This survey provides the first comprehensive review of AVI through the lens of large foundation models.
  arXiv  Detail & Related papers  (2026-05-05T17:59:16Z)
• SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation [15.901895888187711]
  BinauralVGGSound is the first large-scale video-binaural audio dataset designed to support spatially aware video-to-audio generation.<n>Our framework incorporates a visual-guided audio spatialization module that ensures the generated audio exhibits realistic spatial attributes and layered spatial depth.
  arXiv  Detail & Related papers  (2026-01-21T14:14:37Z)
• The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models [17.675850481660863]
  We introduce a hierarchical framework for Auditory Scene Analysis (ASA)<n> guided by this framework, we introduce a system that enables models like Qwen2-Audio to understand and reason about the complex acoustic world.<n>Our work provides a clear pathway for leveraging the powerful reasoning abilities of large models towards holistic acoustic scene analysis.
  arXiv  Detail & Related papers  (2026-01-06T11:54:47Z)
• Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound [5.591620304505415]
  This work presents the first formal framework for Audio-Visual World Models (AVWM)<n>It formulates multimodal environment simulation as a partially observable decision process with audio-visual observations, fine-grained actions, and task rewards.<n>We propose an Audio-Visual Conditional Transformer with a novel modality expert architecture that balances visual and auditory learning.
  arXiv  Detail & Related papers  (2025-11-30T13:11:56Z)
• SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models [62.14165748145729]
  We introduce SPUR, a lightweight, plug-in approach that equips large audio-speaker models with spatial perception.<n>SPUR consists of: (i) a First-Order Ambisonics (FOA) encoder that maps channels to rotation-aware, listener-centric spatial features, integrated into target LALMs via a multimodal adapter; and (ii) SPUR-Set, a spatial QA dataset combining open-source FOA recordings with controlled simulations, emphasizing relative direction, elevation, distance, and overlap for supervised spatial reasoning.
  arXiv  Detail & Related papers  (2025-11-10T01:29:26Z)
• Mixture-of-Experts Framework for Field-of-View Enhanced Signal-Dependent Binauralization of Moving Talkers [3.5522191686718725]
  We propose a novel mixture of experts framework for field-of-view enhancement in signal matching.<n>Our approach enables dynamic spatial audio rendering that adapts to continuous talker motion, allowing users to emphasize or suppress sounds from selected directions.<n>This allows for realtime tracking and enhancement of moving sound sources, supporting applications such as speech focus, noise reduction, and world-locked audio in augmented and virtual reality.
  arXiv  Detail & Related papers  (2025-09-16T21:30:06Z)
• In-the-wild Audio Spatialization with Flexible Text-guided Localization [37.60344400859993]
  To enhance immersive experiences, audio offers spatial awareness of sounding objects in AR, VR, and embodied AI applications.<n>While existing audio spatialization methods can generally map any available monaural audio to audio signals, they often lack the flexible and interactive control needed in complex multi-object user-interactive environments.<n>We propose a Text-guided Audio Spatialization (TAS) framework that utilizes flexible text prompts and evaluates our model from unified generation and comprehension perspectives.
  arXiv  Detail & Related papers  (2025-06-01T09:41:56Z)
• ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling [57.1025908604556]
  An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment. We propose active acoustic sampling, a new task for efficiently building an environment acoustic model of an unmapped environment. We introduce ActiveRIR, a reinforcement learning policy that leverages information from audio-visual sensor streams to guide agent navigation and determine optimal acoustic data sampling positions.
  arXiv  Detail & Related papers  (2024-04-24T21:30:01Z)
• Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark [65.79402756995084]
  Real Acoustic Fields (RAF) is a new dataset that captures real acoustic room data from multiple modalities. RAF is the first dataset to provide densely captured room acoustic data.
  arXiv  Detail & Related papers  (2024-03-27T17:59:56Z)
• AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis [61.07542274267568]
  We study a new task -- real-world audio-visual scene synthesis -- and a first-of-its-kind NeRF-based approach for multimodal learning. We propose an acoustic-aware audio generation module that integrates prior knowledge of audio propagation into NeRF. We present a coordinate transformation module that expresses a view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields.
  arXiv  Detail & Related papers  (2023-02-04T04:17:19Z)
• Few-Shot Audio-Visual Learning of Environment Acoustics [89.16560042178523]
  Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener. We explore how to infer RIRs based on a sparse set of images and echoes observed in the space. In experiments using a state-of-the-art audio-visual simulator for 3D environments, we demonstrate that our method successfully generates arbitrary RIRs.
  arXiv  Detail & Related papers  (2022-06-08T16:38:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.