FaceFilter: Audio-visual speech separation using still images

• URL: http://arxiv.org/abs/2005.07074v1
• Date: Thu, 14 May 2020 15:42:31 GMT
• Title: FaceFilter: Audio-visual speech separation using still images
• Authors: Soo-Whan Chung, Soyeon Choe, Joon Son Chung, Hong-Goo Kang
• Abstract summary: This paper aims to separate a target speaker's speech from a mixture of two speakers using a deep audio-visual speech separation network. Unlike previous works that used lip movement on video clips or pre-enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker.
• Score: 41.97445146257419
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audio-visual speech separation network. Unlike previous works that used lip movement on video clips or pre-enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker. In this task, the conditional feature is obtained from facial appearance in cross-modal biometric task, where audio and visual identity representations are shared in latent space. Learnt identities from facial images enforce the network to isolate matched speakers and extract the voices from mixed speech. It solves the permutation problem caused by swapped channel outputs, frequently occurred in speech separation tasks. The proposed method is far more practical than video-based speech separation since user profile images are readily available on many platforms. Also, unlike speaker-aware separation methods, it is applicable on separation with unseen speakers who have never been enrolled before. We show strong qualitative and quantitative results on challenging real-world examples.

Related papers

• JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation [24.2065254076207]
  We introduce a novel method for joint expression and audio-guided talking face generation. Our method can synthesize high-fidelity talking face videos, achieving state-of-the-art facial expression transfer.
  arXiv  Detail & Related papers  (2024-09-18T17:18:13Z)
• Speaker Extraction with Co-Speech Gestures Cue [79.91394239104908]
  We explore the use of co-speech gestures sequence, e.g. hand and body movements, as the speaker cue for speaker extraction. We propose two networks using the co-speech gestures cue to perform attentive listening on the target speaker. The experimental results show that the co-speech gestures cue is informative in associating the target speaker, and the quality of the extracted speech shows significant improvements over the unprocessed mixture speech.
  arXiv  Detail & Related papers  (2022-03-31T06:48:52Z)
• One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning [20.51814865676907]
  It would be much easier to learn a consistent speech style from a specific speaker, which leads to authentic mouth movements. We propose a novel one-shot talking face generation framework by exploring consistent correlations between audio and visual motions from a specific speaker. Thanks to our learned consistent speaking style, our method generates authentic mouth shapes and vivid movements.
  arXiv  Detail & Related papers  (2021-12-06T02:53:51Z)
• Speech2Video: Cross-Modal Distillation for Speech to Video Generation [21.757776580641902]
  Speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries. The challenge mainly lies in disentangling the distinct visual attributes from audio signals. We propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlabelled video inputs.
  arXiv  Detail & Related papers  (2021-07-10T10:27:26Z)
• Streaming Multi-talker Speech Recognition with Joint Speaker Identification [77.46617674133556]
  SURIT employs the recurrent neural network transducer (RNN-T) as the backbone for both speech recognition and speaker identification. We validate our idea on the Librispeech dataset -- a multi-talker dataset derived from Librispeech, and present encouraging results.
  arXiv  Detail & Related papers  (2021-04-05T18:37:33Z)
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency [111.55430893354769]
  Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement.
  arXiv  Detail & Related papers  (2021-01-08T18:25:24Z)
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation [57.68765353264689]
  Speech enhancement and speech separation are two related tasks. Traditionally, these tasks have been tackled using signal processing and machine learning techniques. Deep learning has been exploited to achieve strong performance.
  arXiv  Detail & Related papers  (2020-08-21T17:24:09Z)
• Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam [100.95498268200777]
  SpeakerBeam exploits an adaptation utterance of the target speaker to extract his/her voice characteristics. SpeakerBeam sometimes fails when speakers have similar voice characteristics, such as in same-gender mixtures. We show experimentally that these strategies greatly improve speech extraction performance, especially for same-gender mixtures.
  arXiv  Detail & Related papers  (2020-01-23T05:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.