Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation

• URL: http://arxiv.org/abs/2604.17656v2
• Date: Wed, 22 Apr 2026 20:49:39 GMT
• Title: Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation
• Abstract summary: Video-to-music (V2M) is the fundamental task of creating background music for an input video.<n>We present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content.
• Score: 58.326977744404566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic and stylistic controllability to the end user. In this paper, we present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content. To balance musical fidelity and semantic understanding, Video-Robin integrates autoregressive planning with diffusion-based synthesis. Specifically, an autoregressive module models global structure by semantically aligning visual and textual inputs to produce high-level music latents. These latents are subsequently refined into coherent, high-fidelity music using local Diffusion Transformers. By factoring semantically driven planning into diffusion-based synthesis, Video-Robin enables fine-grained creator control without sacrificing audio realism. Our proposed model outperforms baselines that solely accept video input and additional feature conditioned baselines on both in-distribution and out-of-distribution benchmarks with a 2.21x speed in inference compared to SOTA. We will open-source everything upon paper acceptance.

Related papers

• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling [59.97930201616015]
  We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone.<n>A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space.<n> Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality.
  arXiv  Detail & Related papers  (2026-04-26T07:48:47Z)
• YingVideo-MV: Music-Driven Multi-Stage Video Generation [22.89609000437466]
  We present YingVideo-MV, the first cascaded framework for music-driven long-video generation.<n>Our approach integrates audio semantic analysis, an interpretable shot planning module, temporal-aware diffusion Transformer architectures.<n>We construct a large-scale Music-in-the-Wild dataset to support the achievement of diverse, high-quality results.
  arXiv  Detail & Related papers  (2025-12-02T07:31:19Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [47.14083940177122]
  ThinkSound is a novel framework that enables stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement, and targeted editing.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• Let Your Video Listen to Your Music! [62.27731415767459]
  We propose a novel framework, MVAA, that automatically edits video to align with the rhythm of a given music track.<n>We modularize the task into a two-step process in our MVAA: aligning motion with audio beats, followed by rhythm-aware video editing.<n>This hybrid approach enables adaptation within 10 minutes with one on a single NVIDIA 4090 GPU using CogVideoX-5b-I2V as the backbone.
  arXiv  Detail & Related papers  (2025-06-23T17:52:16Z)
• GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions [13.9134271174972]
  We present General Video-to-Music Generation model (GVMGen) for generating high-related music to the video input.<n>Our model employs hierarchical attentions to extract and align video features with music in both spatial and temporal dimensions.<n>Our method is versatile, capable of generating multi-style music from different video inputs, even in zero-shot scenarios.
  arXiv  Detail & Related papers  (2025-01-17T06:30:11Z)
• MusicFlow: Cascaded Flow Matching for Text Guided Music Generation [53.63948108922333]
  MusicFlow is a cascaded text-to-music generation model based on flow matching. We leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation.
  arXiv  Detail & Related papers  (2024-10-27T15:35:41Z)
• VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos [32.741262543860934]
  We present a framework for learning to generate background music from video inputs. We develop a generative video-music Transformer with a novel semantic video-music alignment scheme. New temporal video encoder architecture allows us to efficiently process videos consisting of many densely sampled frames.
  arXiv  Detail & Related papers  (2024-09-11T17:56:48Z)
• Compositional Video Generation as Flow Equalization [72.88137795439407]
  Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated unprecedented capability to transform natural language descriptions into stunning and photorealistic videos. Despite the promising results, these models struggle to fully grasp complex compositional interactions between multiple concepts and actions. We introduce bftextVico, a generic framework for compositional video generation that explicitly ensures all concepts are represented properly.
  arXiv  Detail & Related papers  (2024-06-10T16:27:47Z)
• Dance Any Beat: Blending Beats with Visuals in Dance Video Generation [12.018432669719742]
  We introduce a novel task: generating dance videos directly from images of individuals guided by music.<n>Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos.<n>We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment.
  arXiv  Detail & Related papers  (2024-05-15T11:33:07Z)
• Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model [32.801213106782335]
  We develop a generative music AI framework, Video2Music, that can match a provided video. In a thorough experiment, we show that our proposed framework can generate music that matches the video content in terms of emotion.
  arXiv  Detail & Related papers  (2023-11-02T03:33:00Z)
• V2Meow: Meowing to the Visual Beat via Video-to-Music Generation [47.076283429992664]
  V2Meow is a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames.
  arXiv  Detail & Related papers  (2023-05-11T06:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.