SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models

• URL: http://arxiv.org/abs/2405.00878v1
• Date: Wed, 1 May 2024 21:43:57 GMT
• Title: SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models
• Authors: Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakaş, Duygu Ceylan, Erkut Erdem, Aykut Erdem,
• Abstract summary: We propose a method to enable audio-conditioning in large scale image diffusion models. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods.
• Score: 21.669044026456557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjuction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.

Related papers

• Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization [70.13218512896032]
  Generation of audio from text prompts is an important aspect of such processes in the music and film industry. Our hypothesis is focusing on how these aspects of audio generation could improve audio generation performance in the presence of limited data. We synthetically create a preference dataset where each prompt has a winner audio output and some loser audio outputs for the diffusion model to learn from.
  arXiv  Detail & Related papers  (2024-04-15T17:31:22Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion [84.138804145918]
  Deep generative models can generate high-fidelity audio conditioned on various types of representations. These models are prone to generate audible artifacts when the conditioning is flawed or imperfect. We propose a high-fidelity multi-band diffusion-based framework that generates any type of audio modality from low-bitrate discrete representations.
  arXiv  Detail & Related papers  (2023-08-02T22:14:29Z)
• Align, Adapt and Inject: Sound-guided Unified Image Generation [50.34667929051005]
  We propose a unified framework 'Align, Adapt, and Inject' (AAI) for sound-guided image generation, editing, and stylization. Our method adapts input sound into a sound token, like an ordinary word, which can plug and play with existing Text-to-Image (T2I) models. Our proposed AAI outperforms other text and sound-guided state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-20T12:50:49Z)
• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models [50.42886595228255]
  We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining model.
  arXiv  Detail & Related papers  (2023-06-16T05:42:01Z)
• AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation [89.63430567887718]
  We propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings. Using a pre-trained audio encoding model, the proposed method encodes audio into a new token, which can be considered as an adaptation layer between the audio and text representations.
  arXiv  Detail & Related papers  (2023-05-22T14:02:44Z)
• Speech Driven Video Editing via an Audio-Conditioned Diffusion Model [1.6763474728913939]
  We propose a method for end-to-end speech-driven video editing using a denoising diffusion model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing.
  arXiv  Detail & Related papers  (2023-01-10T12:01:20Z)
• TransFusion: Transcribing Speech with Multinomial Diffusion [20.165433724198937]
  We propose a new way to perform speech recognition using a diffusion model conditioned on pretrained speech features. We demonstrate comparable performance to existing high-performing contrastive models on the LibriSpeech speech recognition benchmark. We also propose new techniques for effectively sampling and decoding multinomial diffusion models.
  arXiv  Detail & Related papers  (2022-10-14T10:01:43Z)
• Sound-Guided Semantic Image Manipulation [19.01823634838526]
  We propose a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space. Our method can mix different modalities, i.e., text and audio, which enrich the variety of the image modification. The experiments on zero-shot audio classification and semantic-level image classification show that our proposed model outperforms other text and sound-guided state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-30T13:30:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.