Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

• URL: http://arxiv.org/abs/2608.00463v1
• Date: Sat, 01 Aug 2026 06:04:34 GMT
• Title: Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
• Abstract summary: 3D Gaussian Splatting (3DGS) turns captured or generated imagery into 3D world simulations that users can freely explore, yet these worlds remain silent.<n>Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves.<n>We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding.<n>We present Scene2Sound, a training-free framework built on this grounding.
• Score: 51.55297978842271
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian Splatting (3DGS) turns captured or generated imagery into photorealistic 3D world simulations that users can freely explore, yet these worlds remain silent. Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves. We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding, identifying which objects in the world should emit sound and anchoring each to a persistent 3D position, and present Scene2Sound, a training-free framework built on this grounding. From the input world alone, our pipeline selects viewpoints that jointly cover the scene, identifies sound-emitting objects with a vision-language model, and associates the multi-view detections into 3D instances through Gaussian set matching, which measures the overlap between the Gaussian sets that render each detection. Each source then receives generated audio that a standard object-based audio engine spatializes in real time at arbitrary listener poses. We further propose two spatial-consistency metrics, one testing whether rendered audio responds consistently to listener motion and one testing whether the claimed sources are supported by views held out from their placement. On a curated set of generated 3DGS worlds and on 3DGS scenes generated from real-world 360-degree captures, Scene2Sound preserves the audio quality of strong per-viewpoint baselines while remaining spatially consistent where per-viewpoint and single-panorama pipelines do not, and a user study confirms the perceptual benefit. Project page: https://masaki-lmd.github.io/scene2sound/.

Related papers

• SonoWorld: From One Image to a 3D Audio-Visual Scene [29.64658395133738]
  We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image.<n>We present SonoWorld, the first framework to tackle this challenge.<n>We also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation.
  arXiv  Detail & Related papers  (2026-03-30T17:57:47Z)
• Visual Acoustic Fields [39.43953430861896]
  We propose Visual Acoustic Fields, a framework that bridges hitting sounds and visual signals within a 3D space.<n>Our approach features two key modules: sound generation and sound localization.<n>To the best of our knowledge, this is the first dataset to connect visual and acoustic signals in a 3D context.
  arXiv  Detail & Related papers  (2025-03-31T16:16:10Z)
• SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Camera [61.642416712939095]
  SoundLoc3D treats the task as a set prediction problem, each element in the set corresponds to a potential sound source.<n>We demonstrate the efficiency and superiority of SoundLoc3D on large-scale simulated dataset.
  arXiv  Detail & Related papers  (2024-12-22T05:04:17Z)
• 3D Audio-Visual Segmentation [52.34970001474347]
  Recognizing the sounding objects in scenes is a longstanding objective in embodied AI, with diverse applications in robotics and AR/VR/MR.<n>We propose a new approach, EchoSegnet, characterized by integrating the ready-to-use knowledge from pretrained 2D audio-visual foundation models.<n>Experiments demonstrate that EchoSegnet can effectively segment sounding objects in 3D space on our new benchmark, representing a significant advancement in the field of embodied AI.
  arXiv  Detail & Related papers  (2024-11-04T16:30:14Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene.<n>Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio.<n>We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment.<n>Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning [127.1119359047849]
  We introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio rendering for 3D environments. It generates highly realistic acoustics for arbitrary sounds captured from arbitrary microphone locations. SoundSpaces 2.0 is publicly available to facilitate wider research for perceptual systems that can both see and hear.
  arXiv  Detail & Related papers  (2022-06-16T17:17:44Z)
• Learning to Set Waypoints for Audio-Visual Navigation [89.42192208471735]
  In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source. Existing models learn to act at a fixed granularity of agent motion and rely on simple recurrent aggregations of the audio observations. We introduce a reinforcement learning approach to audio-visual navigation with two key novel elements.
  arXiv  Detail & Related papers  (2020-08-21T18:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.