A Comprehensive Survey on Generative AI for Video-to-Music Generation

• URL: http://arxiv.org/abs/2502.12489v1
• Date: Tue, 18 Feb 2025 03:18:54 GMT
• Title: A Comprehensive Survey on Generative AI for Video-to-Music Generation
• Authors: Shulei Ji, Songruoyao Wu, Zihao Wang, Shuyu Li, Kejun Zhang,
• Abstract summary: This paper presents a comprehensive review of video-to-music generation using deep generative AI techniques.<n>We focus on three key components: visual feature extraction, music generation frameworks, and conditioning mechanisms.
• Score: 15.575851379886952
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The burgeoning growth of video-to-music generation can be attributed to the ascendancy of multimodal generative models. However, there is a lack of literature that comprehensively combs through the work in this field. To fill this gap, this paper presents a comprehensive review of video-to-music generation using deep generative AI techniques, focusing on three key components: visual feature extraction, music generation frameworks, and conditioning mechanisms. We categorize existing approaches based on their designs for each component, clarifying the roles of different strategies. Preceding this, we provide a fine-grained classification of video and music modalities, illustrating how different categories influence the design of components within the generation pipelines. Furthermore, we summarize available multimodal datasets and evaluation metrics while highlighting ongoing challenges in the field.

Related papers

• Extending Visual Dynamics for Video-to-Music Generation [51.274561293909926]
  DyViM is a novel framework to enhance dynamics modeling for video-to-music generation. High-level semantics are conveyed through a cross-attention mechanism. Experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-04-10T09:47:26Z)
• A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives [14.69952700449563]
  Multi-modal music generation is an emerging research area with broad applications. This paper reviews this field, categorizing music generation systems from the perspective of modalities. Key challenges in this area include effective multi-modal integration, large-scale comprehensive datasets, and systematic evaluation methods.
  arXiv  Detail & Related papers  (2025-04-01T14:26:25Z)
• Vision-to-Music Generation: A Survey [10.993775589904251]
  Vision-to-music generation shows vast application prospects in fields such as film scoring, short video creation, and dance music synthesis. Research in vision-to-music is still in its preliminary stage due to its complex internal structure and the difficulty of modeling dynamic relationships with video. Existing surveys focus on general music generation without comprehensive discussion on vision-to-music.
  arXiv  Detail & Related papers  (2025-03-27T08:21:54Z)
• ASurvey: Spatiotemporal Consistency in Video Generation [72.82267240482874]
  Video generation schemes by leveraging a dynamic visual generation method, pushes the boundaries of Artificial Intelligence Generated Content (AIGC) Recent works have aimed at addressing thetemporal consistency issue in video generation, while few literature review has been organized from this perspective. We systematically review recent advances in video generation, covering five key aspects: foundation models, information representations, generation schemes, post-processing techniques, and evaluation metrics.
  arXiv  Detail & Related papers  (2025-02-25T05:20:51Z)
• GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions [13.9134271174972]
  We present General Video-to-Music Generation model (GVMGen) for generating high-related music to the video input.<n>Our model employs hierarchical attentions to extract and align video features with music in both spatial and temporal dimensions.<n>Our method is versatile, capable of generating multi-style music from different video inputs, even in zero-shot scenarios.
  arXiv  Detail & Related papers  (2025-01-17T06:30:11Z)
• Video-to-Audio Generation with Hidden Alignment [27.11625918406991]
  We offer insights into the video-to-audio generation paradigm, focusing on vision encoders, auxiliary embeddings, and data augmentation techniques. We demonstrate our model exhibits state-of-the-art video-to-audio generation capabilities.
  arXiv  Detail & Related papers  (2024-07-10T08:40:39Z)
• TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation [97.96178992465511]
  We argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics.
  arXiv  Detail & Related papers  (2024-06-12T21:41:32Z)
• LLMs Meet Multimodal Generation and Editing: A Survey [89.76691959033323]
  This survey elaborates on multimodal generation and editing across various domains, comprising image, video, 3D, and audio. We summarize the notable advancements with milestone works in these fields and categorize these studies into LLM-based and CLIP/T5-based methods. We dig into tool-augmented multimodal agents that can leverage existing generative models for human-computer interaction.
  arXiv  Detail & Related papers  (2024-05-29T17:59:20Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• Multi-object Video Generation from Single Frame Layouts [84.55806837855846]
  We propose a video generative framework capable of synthesizing global scenes with local objects. Our framework is a non-trivial adaptation from image generation methods, and is new to this field. Our model has been evaluated on two widely-used video recognition benchmarks.
  arXiv  Detail & Related papers  (2023-05-06T09:07:01Z)
• A Review of Intelligent Music Generation Systems [4.287960539882345]
  ChatGPT has significantly reduced the barrier to entry for non-professionals in creative endeavors. Modern generative algorithms can extract patterns implicit in a piece of music based on rule constraints or a musical corpus.
  arXiv  Detail & Related papers  (2022-11-16T13:43:16Z)
• A Survey on Deep Learning Technique for Video Segmentation [147.0767454918527]
  Video segmentation plays a critical role in a broad range of practical applications. Deep learning based approaches have been dedicated to video segmentation and delivered compelling performance.
  arXiv  Detail & Related papers  (2021-07-02T15:51:07Z)
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions [10.179835761549471]
  This paper attempts to provide an overview of various composition tasks under different music generation levels using deep learning. In addition, we summarize datasets suitable for diverse tasks, discuss the music representations, the evaluation methods as well as the challenges under different levels, and finally point out several future directions.
  arXiv  Detail & Related papers  (2020-11-13T08:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.