Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

• URL: http://arxiv.org/abs/2301.04474v3
• Date: Thu, 11 May 2023 11:56:42 GMT
• Title: Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
• Authors: Dan Bigioi, Shubhajit Basak, Micha{\l} Stypu{\l}kowski, Maciej Zi\k{e}ba, Hugh Jordan, Rachel McDonnell, Peter Corcoran
• Abstract summary: We propose a method for end-to-end speech-driven video editing using a denoising diffusion model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing.
• Score: 1.6763474728913939
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Taking inspiration from recent developments in visual generative tasks using diffusion models, we propose a method for end-to-end speech-driven video editing using a denoising diffusion model. Given a video of a talking person, and a separate auditory speech recording, the lip and jaw motions are re-synchronized without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. Proof of concept results are demonstrated on both single-speaker and multi-speaker video editing, providing a baseline model on the CREMA-D audiovisual data set. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing.

Related papers

• SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models [21.669044026456557]
  We propose a method to enable audio-conditioning in large scale image diffusion models. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods.
  arXiv  Detail & Related papers  (2024-05-01T21:43:57Z)
• Synthesizing Audio from Silent Video using Sequence to Sequence Modeling [0.0]
  We propose a novel method to generate audio from video using a sequence-to-sequence model. Our approach employs a 3D Vector Quantized Variational Autoencoder (VQ-VAE) to capture the video's spatial and temporal structures. Our model aims to enhance applications like CCTV footage analysis, silent movie restoration, and video generation models.
  arXiv  Detail & Related papers  (2024-04-25T22:19:42Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models [57.71199494492223]
  We propose a Unified Speech Enhancement and Editing (uSee) model with conditional diffusion models to handle various tasks at the same time in a generative manner. Our experiments show that our proposed uSee model can achieve superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models.
  arXiv  Detail & Related papers  (2023-10-02T04:36:39Z)
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI [64.71397830291838]
  Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction. With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys.
  arXiv  Detail & Related papers  (2023-03-23T15:17:15Z)
• VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation [88.49030739715701]
  This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation.
  arXiv  Detail & Related papers  (2023-03-15T02:16:39Z)
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation [70.74377373885645]
  We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously. MM-Diffusion consists of a sequential multi-modal U-Net for a joint denoising process by design. Experiments show superior results in unconditional audio-video generation, and zero-shot conditional tasks.
  arXiv  Detail & Related papers  (2022-12-19T14:11:52Z)
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks [54.43697805589634]
  We propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) Our model consists of an encoder-decoder architecture that receives raw video as input and generates speech. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID.
  arXiv  Detail & Related papers  (2021-04-27T17:12:30Z)
• Audiovisual Saliency Prediction in Uncategorized Video Sequences based on Audio-Video Correlation [0.0]
  This work aims to provide a generic audio/video saliency model augmenting a visual saliency map with an audio saliency map computed by synchronizing low-level audio and visual features. The proposed model was evaluated using different criteria against eye fixations data for a publicly available DIEM video dataset.
  arXiv  Detail & Related papers  (2021-01-07T14:22:29Z)
• Audio-visual Speech Separation with Adversarially Disentangled Visual Representation [23.38624506211003]
  Speech separation aims to separate individual voice from an audio mixture of multiple simultaneous talkers. In our model, we use the face detector to detect the number of speakers in the scene and use visual information to avoid the permutation problem. Our proposed model is shown to outperform the state-of-the-art audio-only model and three audio-visual models.
  arXiv  Detail & Related papers  (2020-11-29T10:48:42Z)
• Speech Prediction in Silent Videos using Variational Autoencoders [29.423462898526605]
  We present a model for generating speech in a silent video. The proposed model combines recurrent neural networks and variational deep generative models to learn the auditory's conditional distribution. We demonstrate the performance of our model on the GRID dataset based on standard benchmarks.
  arXiv  Detail & Related papers  (2020-11-14T17:09:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.