SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field

• URL: http://arxiv.org/abs/2406.11006v1
• Date: Sun, 16 Jun 2024 16:40:26 GMT
• Title: SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field
• Authors: Yuhang He, Shitong Xu, Jia-Xing Zhong, Sangyun Shin, Niki Trigoni, Andrew Markham,
• Abstract summary: SPEAR is a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction. We show SPEAR superiority on both synthetic, photo-realistic and real-world dataset.
• Score: 39.19609821736598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present SPEAR, a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction in an acoustic 3D space with a single stationary audio source. Unlike traditional source-to-receiver modelling methods that require prior space acoustic properties knowledge to rigorously model audio propagation from source to receiver, we propose to predict by warping the spatial acoustic effects from one reference receiver position to another target receiver position, so that the warped audio essentially accommodates all spatial acoustic effects belonging to the target position. SPEAR can be trained in a data much more readily accessible manner, in which we simply ask two robots to independently record spatial audio at different positions. We further theoretically prove the universal existence of the warping field if and only if one audio source presents. Three physical principles are incorporated to guide SPEAR network design, leading to the learned warping field physically meaningful. We demonstrate SPEAR superiority on both synthetic, photo-realistic and real-world dataset, showing the huge potential of SPEAR to various down-stream robotic tasks.

Related papers

• ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model [2.2927722373373247]
  We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects.
  arXiv  Detail & Related papers  (2024-10-19T02:28:53Z)
• Blind Spatial Impulse Response Generation from Separate Room- and Scene-Specific Information [0.42970700836450487]
  knowledge of the users' real acoustic environment is crucial for rendering virtual sounds that seamlessly blend into the environment. We show how both room- and position-specific parameters are considered in the final output.
  arXiv  Detail & Related papers  (2024-09-23T12:41:31Z)
• SOAF: Scene Occlusion-aware Neural Acoustic Field [9.651041527067907]
  We propose a new approach called Scene Occlusion-aware Acoustic Field (SOAF) for accurate sound generation. Our approach derives a prior for sound energy field using distance-aware parametric sound-propagation modelling. We extract features from local acoustic field centred around the receiver using a Fibonacci Sphere to generate audio for novel views.
  arXiv  Detail & Related papers  (2024-07-02T13:40:56Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio. We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment. Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios [36.50731790624643]
  We introduce RIR-SF, a novel spatial feature based on room impulse response (RIR) RIR-SF significantly outperforms traditional 3D spatial features, showing superior theoretical and empirical performance. We also propose an optimized all-neural multi-channel ASR framework for RIR-SF, achieving a relative 21.3% reduction in CER for target speaker ASR in multi-channel settings.
  arXiv  Detail & Related papers  (2023-10-31T20:42:08Z)
• Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes [69.03289331433874]
  We present an end-to-end audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based sound propagation method to generate acoustic effects for 3D models of real environments.
  arXiv  Detail & Related papers  (2023-02-02T04:09:23Z)
• Implicit Neural Spatial Filtering for Multichannel Source Separation in the Waveform Domain [131.74762114632404]
  The model is trained end-to-end and performs spatial processing implicitly. We evaluate the proposed model on a real-world dataset and show that the model matches the performance of an oracle beamformer.
  arXiv  Detail & Related papers  (2022-06-30T17:13:01Z)
• Few-Shot Audio-Visual Learning of Environment Acoustics [89.16560042178523]
  Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener. We explore how to infer RIRs based on a sparse set of images and echoes observed in the space. In experiments using a state-of-the-art audio-visual simulator for 3D environments, we demonstrate that our method successfully generates arbitrary RIRs.
  arXiv  Detail & Related papers  (2022-06-08T16:38:24Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.