Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition

• URL: http://arxiv.org/abs/2410.03335v2
• Date: Tue, 14 Jan 2025 11:59:03 GMT
• Title: Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition
• Authors: Zixuan Wang, Chi-Keung Tang, Yu-Wing Tai,
• Abstract summary: We introduce Audio-Agent, a framework for audio generation, editing and composition based on text or video inputs. In our method, we utilize a pre-trained TTA diffusion network as the audio generation agent to work in tandem with GPT-4. For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with the generated audio.
• Score: 72.22243595269389
• License:
• Abstract: We introduce Audio-Agent, a multimodal framework for audio generation, editing and composition based on text or video inputs. Conventional approaches for text-to-audio (TTA) tasks often make single-pass inferences from text descriptions. While straightforward, this design struggles to produce high-quality audio when given complex text conditions. In our method, we utilize a pre-trained TTA diffusion network as the audio generation agent to work in tandem with GPT-4, which decomposes the text condition into atomic, specific instructions and calls the agent for audio generation. In doing so, Audio-Agent can generate high-quality audio that is closely aligned with the provided text or video exhibiting complex and multiple events, while supporting variable-length and variable-volume generation. For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with the generated audio, a process that can be tedious and time-consuming. Instead, we propose a simpler approach by fine-tuning a pre-trained Large Language Model (LLM), e.g., Gemma2-2B-it, to obtain both semantic and temporal conditions that bridge the video and audio modality. Consequently, our framework contributes a comprehensive solution for both TTA and VTA tasks without substantial computational overhead in training.

Related papers

• VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation [27.9571263633586]
  We introduce VinTAGe, a flow-based transformer model that jointly considers text and video to guide audio generation. Our framework comprises two key components: a Visual-Text and a Joint VT-SiT model. Due to the lack of appropriate benchmarks, we also introduce VinTAGe-Bench, a dataset of 636 video-text-audio pairs containing both onscreen and offscreen sounds.
  arXiv  Detail & Related papers  (2024-12-14T09:36:10Z)
• Tell What You Hear From What You See -- Video to Audio Generation Through Text [17.95017332858846]
  VATT is a multi-modal generative framework that takes a video and an optional text prompt as input, and generates audio and optional textual description of the audio. VATT enables controllable video-to-audio generation through text as well as suggesting text prompts for videos through audio captions.
  arXiv  Detail & Related papers  (2024-11-08T16:29:07Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)
• Text-to-Audio Generation Synchronized with Videos [44.848393652233796]
  We introduce a groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench. We also present a simple yet effective video-aligned TTA generation model, namely T2AV. It employs a temporal multi-head attention transformer to extract and understand temporal nuances from video data, a feat amplified by our Audio-Visual ControlNet.
  arXiv  Detail & Related papers  (2024-03-08T22:27:38Z)
• Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning [50.28566759231076]
  We propose an innovative, automatic approach to establish an audio dataset with high-quality captions. Specifically, we construct a large-scale, high-quality, audio-language dataset, named as Auto-ACD, comprising over 1.5M audio-text pairs. We employ LLM to paraphrase a congruent caption for each audio, guided by the extracted multi-modality clues.
  arXiv  Detail & Related papers  (2023-09-20T17:59:32Z)
• Retrieval-Augmented Text-to-Audio Generation [36.328134891428085]
  We show that the state-of-the-art models, such as AudioLDM, are biased in their generation performance. We propose a simple retrieval-augmented approach for TTA models. We show that Re-AudioLDM can generate realistic audio for complex scenes, rare audio classes, and even unseen audio types.
  arXiv  Detail & Related papers  (2023-09-14T22:35:39Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)
• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models [50.42886595228255]
  We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining model.
  arXiv  Detail & Related papers  (2023-06-16T05:42:01Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data [15.658471125219224]
  Multimodal pre-training for audio-and-text has been proven to be effective and has significantly improved the performance of many downstream speech understanding tasks. However, these state-of-the-art pre-training audio-text models work well only when provided with large amount of parallel audio-and-text data. In this paper, we investigate whether it is possible to pre-train an audio-text model with low-resource parallel data.
  arXiv  Detail & Related papers  (2022-04-10T10:25:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.