RespVAD: Voice Activity Detection via Video-Extracted Respiration Patterns

• URL: http://arxiv.org/abs/2008.09466v1
• Date: Fri, 21 Aug 2020 13:26:24 GMT
• Title: RespVAD: Voice Activity Detection via Video-Extracted Respiration Patterns
• Authors: Arnab Kumar Mondal, Prathosh A.P
• Abstract summary: Voice Activity Detection (VAD) refers to the task of identification of regions of human speech in digital signals such as audio and video. respiration forms the primary source of energy for speech production. An audio-independent VAD technique using the respiration pattern extracted from the speakers' video is developed.
• Score: 5.716047866174048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Voice Activity Detection (VAD) refers to the task of identification of regions of human speech in digital signals such as audio and video. While VAD is a necessary first step in many speech processing systems, it poses challenges when there are high levels of ambient noise during the audio recording. To improve the performance of VAD in such conditions, several methods utilizing the visual information extracted from the region surrounding the mouth/lip region of the speakers' video recording have been proposed. Even though these provide advantages over audio-only methods, they depend on faithful extraction of lip/mouth regions. Motivated by these, a new paradigm for VAD based on the fact that respiration forms the primary source of energy for speech production is proposed. Specifically, an audio-independent VAD technique using the respiration pattern extracted from the speakers' video is developed. The Respiration Pattern is first extracted from the video focusing on the abdominal-thoracic region of a speaker using an optical flow based method. Subsequently, voice activity is detected from the respiration pattern signal using neural sequence-to-sequence prediction models. The efficacy of the proposed method is demonstrated through experiments on a challenging dataset recorded in real acoustic environments and compared with four previous methods based on audio and visual cues.

Related papers

• CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection [2.110168344647122]
  Voice Activity Detection (VAD) is the process of automatically determining whether a person is speaking and identifying the timing of their speech. We introduce a novel approach leveraging Contrastive Language-Image Pretraining (CLIP) models. Our approach outperforms several audio-visual methods despite its simplicity, and without requiring pre-training on extensive audio-visual datasets.
  arXiv  Detail & Related papers  (2024-10-18T14:43:34Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene. Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio. We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment. Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• Egocentric Audio-Visual Noise Suppression [11.113020254726292]
  This paper studies audio-visual noise suppression for egocentric videos. Video camera emulates off-screen speaker's view of the outside world. We first demonstrate that egocentric visual information is helpful for noise suppression.
  arXiv  Detail & Related papers  (2022-11-07T15:53:12Z)
• Audio-visual speech enhancement with a deep Kalman filter generative model [0.0]
  We present an audiovisual deep Kalman filter (AV-DKF) generative model which assumes a first-order Markov chain model for the latent variables. We develop an efficient inference methodology to estimate speech signals at test time.
  arXiv  Detail & Related papers  (2022-11-02T09:50:08Z)
• AVATAR: Unconstrained Audiovisual Speech Recognition [75.17253531162608]
  We propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) trained end-to-end from spectrograms and full-frame RGB. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise. We also create a new, real-world test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.
  arXiv  Detail & Related papers  (2022-06-15T17:33:19Z)
• Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos [101.83513408195692]
  We propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video. The proposed method outperforms 12 state-of-the-art saliency prediction methods, and achieves competitive results in sound source localization.
  arXiv  Detail & Related papers  (2021-11-05T14:35:08Z)
• Learning Audio-Visual Dereverberation [87.52880019747435]
  Reverberation from audio reflecting off surfaces and objects in the environment not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Our idea is to learn to dereverberate speech from audio-visual observations. We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed sounds and visual scene.
  arXiv  Detail & Related papers  (2021-06-14T20:01:24Z)
• DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion [51.83469048737548]
  We propose DiffSVC, an SVC system based on denoising diffusion probabilistic model. A denoising module is trained in DiffSVC, which takes destroyed mel spectrogram and its corresponding step information as input to predict the added Gaussian noise. Experiments show that DiffSVC can achieve superior conversion performance in terms of naturalness and voice similarity to current state-of-the-art SVC approaches.
  arXiv  Detail & Related papers  (2021-05-28T14:26:40Z)
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation [57.68765353264689]
  Speech enhancement and speech separation are two related tasks. Traditionally, these tasks have been tackled using signal processing and machine learning techniques. Deep learning has been exploited to achieve strong performance.
  arXiv  Detail & Related papers  (2020-08-21T17:24:09Z)
• Deep Variational Generative Models for Audio-visual Speech Separation [33.227204390773316]
  We propose an unsupervised technique based on audio-visual generative modeling of clean speech. To better utilize the visual information, the posteriors of the latent variables are inferred from mixed speech. Our experiments show that the proposed unsupervised VAE-based method yields better separation performance than NMF-based approaches.
  arXiv  Detail & Related papers  (2020-08-17T10:12:33Z)
• How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition [10.74796391075403]
  This study investigates the inner workings of AV Align and visualises the audio-visual alignment patterns. We find that AV Align learns to align acoustic and visual representations of speech at the frame level on TCD-TIMIT in a generally monotonic pattern. We propose a regularisation method which involves predicting lip-related Action Units from visual representations.
  arXiv  Detail & Related papers  (2020-04-17T13:59:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.