RevRIR: Joint Reverberant Speech and Room Impulse Response Embedding using Contrastive Learning with Application to Room Shape Classification

• URL: http://arxiv.org/abs/2406.03120v1
• Date: Wed, 5 Jun 2024 10:13:55 GMT
• Title: RevRIR: Joint Reverberant Speech and Room Impulse Response Embedding using Contrastive Learning with Application to Room Shape Classification
• Authors: Jacob Bitterman, Daniel Levi, Hilel Hagai Diamandi, Sharon Gannot, Tal Rosenwein,
• Abstract summary: We introduce a dual-encoder architecture that facilitates the estimation of room parameters directly from speech utterances. A contrastive loss encoder function is employed to embed the speech and the acoustic response jointly. In the test phase, only the reverberant utterance is available, and its embedding is used for the task of room shape classification.
• Score: 8.90841350214225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on room fingerprinting, a task involving the analysis of an audio recording to determine the specific volume and shape of the room in which it was captured. While it is relatively straightforward to determine the basic room parameters from the Room Impulse Responses (RIR), doing so from a speech signal is a cumbersome task. To address this challenge, we introduce a dual-encoder architecture that facilitates the estimation of room parameters directly from speech utterances. During pre-training, one encoder receives the RIR while the other processes the reverberant speech signal. A contrastive loss function is employed to embed the speech and the acoustic response jointly. In the fine-tuning stage, the specific classification task is trained. In the test phase, only the reverberant utterance is available, and its embedding is used for the task of room shape classification. The proposed scheme is extensively evaluated using simulated acoustic environments.

Related papers

• Blind Spatial Impulse Response Generation from Separate Room- and Scene-Specific Information [0.42970700836450487]
  knowledge of the users' real acoustic environment is crucial for rendering virtual sounds that seamlessly blend into the environment. We show how both room- and position-specific parameters are considered in the final output.
  arXiv  Detail & Related papers  (2024-09-23T12:41:31Z)
• Speech Rhythm-Based Speaker Embeddings Extraction from Phonemes and Phoneme Duration for Multi-Speaker Speech Synthesis [16.497022070614236]
  This paper proposes a speech rhythm-based method for speaker embeddings to model phoneme duration using a few utterances by the target speaker. A novel feature of the proposed method is the rhythm-based embeddings extracted from phonemes and their durations, which are known to be related to speaking rhythm.
  arXiv  Detail & Related papers  (2024-02-11T02:26:43Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• Few-Shot Audio-Visual Learning of Environment Acoustics [89.16560042178523]
  Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener. We explore how to infer RIRs based on a sparse set of images and echoes observed in the space. In experiments using a state-of-the-art audio-visual simulator for 3D environments, we demonstrate that our method successfully generates arbitrary RIRs.
  arXiv  Detail & Related papers  (2022-06-08T16:38:24Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• CASA-Based Speaker Identification Using Cascaded GMM-CNN Classifier in Noisy and Emotional Talking Conditions [1.6449390849183358]
  This work aims at intensifying text-independent speaker identification performance in real application situations such as noisy and emotional talking conditions. This research proposes and evaluates a novel algorithm to improve the accuracy of speaker identification in emotional and highly-noise susceptible conditions.
  arXiv  Detail & Related papers  (2021-02-11T08:56:12Z)
• Joint Blind Room Acoustic Characterization From Speech And Music Signals Using Convolutional Recurrent Neural Networks [13.12834490248018]
  Reverberation time, clarity, and direct-to-reverberant ratio are acoustic parameters that have been defined to describe reverberant environments. Recent audio combined with machine learning suggests that one could estimate those parameters blindly using speech or music signals. We propose a robust end-to-end method to achieve blind joint acoustic parameter estimation using speech and/or music signals.
  arXiv  Detail & Related papers  (2020-10-21T17:41:21Z)
• Simultaneous Denoising and Dereverberation Using Deep Embedding Features [64.58693911070228]
  We propose a joint training method for simultaneous speech denoising and dereverberation using deep embedding features. At the denoising stage, the DC network is leveraged to extract noise-free deep embedding features. At the dereverberation stage, instead of using the unsupervised K-means clustering algorithm, another neural network is utilized to estimate the anechoic speech.
  arXiv  Detail & Related papers  (2020-04-06T06:34:01Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.