Blind Spatial Impulse Response Generation from Separate Room- and Scene-Specific Information

• URL: http://arxiv.org/abs/2409.14971v1
• Date: Mon, 23 Sep 2024 12:41:31 GMT
• Title: Blind Spatial Impulse Response Generation from Separate Room- and Scene-Specific Information
• Authors: Francesc Lluís, Nils Meyer-Kahlen,
• Abstract summary: knowledge of the users' real acoustic environment is crucial for rendering virtual sounds that seamlessly blend into the environment. We show how both room- and position-specific parameters are considered in the final output.
• Score: 0.42970700836450487
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: For audio in augmented reality (AR), knowledge of the users' real acoustic environment is crucial for rendering virtual sounds that seamlessly blend into the environment. As acoustic measurements are usually not feasible in practical AR applications, information about the room needs to be inferred from available sound sources. Then, additional sound sources can be rendered with the same room acoustic qualities. Crucially, these are placed at different positions than the sources available for estimation. Here, we propose to use an encoder network trained using a contrastive loss that maps input sounds to a low-dimensional feature space representing only room-specific information. Then, a diffusion-based spatial room impulse response generator is trained to take the latent space and generate a new response, given a new source-receiver position. We show how both room- and position-specific parameters are considered in the final output.

Related papers

• ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model [2.2927722373373247]
  We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects.
  arXiv  Detail & Related papers  (2024-10-19T02:28:53Z)
• SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field [39.19609821736598]
  SPEAR is a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction. We show SPEAR superiority on both synthetic, photo-realistic and real-world dataset.
  arXiv  Detail & Related papers  (2024-06-16T16:40:26Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio. We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment. Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• RevRIR: Joint Reverberant Speech and Room Impulse Response Embedding using Contrastive Learning with Application to Room Shape Classification [8.90841350214225]
  We introduce a dual-encoder architecture that facilitates the estimation of room parameters directly from speech utterances. A contrastive loss encoder function is employed to embed the speech and the acoustic response jointly. In the test phase, only the reverberant utterance is available, and its embedding is used for the task of room shape classification.
  arXiv  Detail & Related papers  (2024-06-05T10:13:55Z)
• ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling [57.1025908604556]
  An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment. We propose active acoustic sampling, a new task for efficiently building an environment acoustic model of an unmapped environment. We introduce ActiveRIR, a reinforcement learning policy that leverages information from audio-visual sensor streams to guide agent navigation and determine optimal acoustic data sampling positions.
  arXiv  Detail & Related papers  (2024-04-24T21:30:01Z)
• Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields [61.07542274267568]
  This letter proposes a novel Neural Acoustic Context Field approach, called NACF, to parameterize an audio scene. Driven by the unique properties of RIR, we design a temporal correlation module and multi-scale energy decay criterion. Experimental results show that NACF outperforms existing field-based methods by a notable margin.
  arXiv  Detail & Related papers  (2023-09-27T19:50:50Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• Few-Shot Audio-Visual Learning of Environment Acoustics [89.16560042178523]
  Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener. We explore how to infer RIRs based on a sparse set of images and echoes observed in the space. In experiments using a state-of-the-art audio-visual simulator for 3D environments, we demonstrate that our method successfully generates arbitrary RIRs.
  arXiv  Detail & Related papers  (2022-06-08T16:38:24Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• Deep Sound Field Reconstruction in Real Rooms: Introducing the ISOBEL Sound Field Dataset [0.0]
  This paper extends evaluations of sound field reconstruction at low frequencies by introducing a dataset with measurements from four real rooms. The paper advances on a recent deep learning-based method for sound field reconstruction using a very low number of microphones.
  arXiv  Detail & Related papers  (2021-02-12T11:34:18Z)
• Joint Blind Room Acoustic Characterization From Speech And Music Signals Using Convolutional Recurrent Neural Networks [13.12834490248018]
  Reverberation time, clarity, and direct-to-reverberant ratio are acoustic parameters that have been defined to describe reverberant environments. Recent audio combined with machine learning suggests that one could estimate those parameters blindly using speech or music signals. We propose a robust end-to-end method to achieve blind joint acoustic parameter estimation using speech and/or music signals.
  arXiv  Detail & Related papers  (2020-10-21T17:41:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.