Array2BR: An End-to-End Noise-immune Binaural Audio Synthesis from Microphone-array Signals

• URL: http://arxiv.org/abs/2410.05739v1
• Date: Tue, 8 Oct 2024 06:55:35 GMT
• Title: Array2BR: An End-to-End Noise-immune Binaural Audio Synthesis from Microphone-array Signals
• Authors: Cheng Chi, Xiaoyu Li, Andong Li, Yuxuan Ke, Xiaodong Li, Chengshi Zheng,
• Abstract summary: This paper proposes a new end-to-end noise-immune synthesis framework from microphone-array signals, abbreviated as Array2BR. Compared with existing methods, the proposed method achieved better performance in terms of both objective and subjective metric scores.
• Score: 31.30005077444649
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Telepresence technology aims to provide an immersive virtual presence for remote conference applications, and it is extremely important to synthesize high-quality binaural audio signals for this aim. Because the ambient noise is often inevitable in practical application scenarios, it is highly desired that binaural audio signals without noise can be obtained from microphone-array signals directly. For this purpose, this paper proposes a new end-to-end noise-immune binaural audio synthesis framework from microphone-array signals, abbreviated as Array2BR, and experimental results show that binaural cues can be correctly mapped and noise can be well suppressed simultaneously using the proposed framework. Compared with existing methods, the proposed method achieved better performance in terms of both objective and subjective metric scores.

Related papers

• HARP: A Large-Scale Higher-Order Ambisonic Room Impulse Response Dataset [0.6568378556428859]
  This contribution introduces a dataset of 7th-order Ambisonic Room Impulse Responses (HOA-RIRs) created using the Image Source Method. By employing higher-order Ambisonics, our dataset enables precise spatial audio reproduction. The presented 64-microphone configuration allows us to capture RIRs directly in the Spherical Harmonics domain.
  arXiv  Detail & Related papers  (2024-11-21T15:16:48Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio. We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment. Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes [69.03289331433874]
  We present an end-to-end audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based sound propagation method to generate acoustic effects for 3D models of real environments.
  arXiv  Detail & Related papers  (2023-02-02T04:09:23Z)
• Inference and Denoise: Causal Inference-based Neural Speech Enhancement [83.4641575757706]
  This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. The proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE.
  arXiv  Detail & Related papers  (2022-11-02T15:03:50Z)
• End-to-End Binaural Speech Synthesis [71.1869877389535]
  We present an end-to-end speech synthesis system that combines a low-bitrate audio system with a powerful decoder. We demonstrate the capability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.
  arXiv  Detail & Related papers  (2022-07-08T05:18:36Z)
• BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis [129.86743102915986]
  We formulate the synthesis process from a different perspective by decomposing the audio into a common part. We propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Experiment results show that BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics.
  arXiv  Detail & Related papers  (2022-05-30T02:09:26Z)
• Visually Informed Binaural Audio Generation without Binaural Audios [130.80178993441413]
  We propose PseudoBinaural, an effective pipeline that is free of recordings. We leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received audios. Our-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference.
  arXiv  Detail & Related papers  (2021-04-13T13:07:33Z)
• Scene-Agnostic Multi-Microphone Speech Dereverberation [47.735158037490834]
  We present an NN architecture that can cope with microphone arrays whose number and positions are unknown. Our approach harnesses recent advances in deep learning on set-structured data to design an architecture that enhances the reverberant log-spectrum.
  arXiv  Detail & Related papers  (2020-10-22T17:13:12Z)
• Learning Joint Articulatory-Acoustic Representations with Normalizing Flows [7.183132975698293]
  We find a joint latent representation between the articulatory and acoustic domain for vowel sounds via invertible neural network models. Our approach achieves both articulatory-to-acoustic as well as acoustic-to-articulatory mapping, thereby demonstrating our success in achieving a joint encoding of both the domains.
  arXiv  Detail & Related papers  (2020-05-16T04:34:36Z)
• Robust Speaker Recognition Using Speech Enhancement And Attention Model [37.33388614967888]
  Instead of individually processing speech enhancement and speaker recognition, the two modules are integrated into one framework by a joint optimisation using deep neural networks. To increase robustness against noise, a multi-stage attention mechanism is employed to highlight the speaker related features learned from context information in time and frequency domain. The obtained results show that the proposed approach using speech enhancement and multi-stage attention models outperforms two strong baselines not using them in most acoustic conditions in our experiments.
  arXiv  Detail & Related papers  (2020-01-14T20:03:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.