Related papers: VIBE: Video Instruction-aligned Background music gEneration

VIBE: Video Instruction-aligned Background music gEneration

URL: http://arxiv.org/abs/2608.30125v1
Date: Mon, 31 Aug 2026 01:19:55 GMT
Title: VIBE: Video Instruction-aligned Background music gEneration
Abstract summary: Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations.<n>We introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages Conditioning Connection.
Score: 58.326977744404566
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction objectives and the representational bottleneck of static cross-modal conditioning in Diffusion Autoregressive (DAR) architectures. To resolve this, we introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages: (1) Conditioning Connection, a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and (2) a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints (e.g., tempo, key) and soft, subjective qualities (e.g., musicality, multimodal alignment) with a structured 5-stage training curriculum. Upon evaluation using audio-visual alignment, instruction following, and audio quality metrics, along with a subjective human evaluation study, we observe that VIBE demonstrates enhanced controllability and instruction adherence while performing comparably to most evaluated baselines on generation fidelity and multimodal alignment.

Related papers

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation [53.47090840868696]
Video-to-audio (V2A) generation extends image-to-audio generation (I2A)<n>We introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A.<n>We show that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality.
arXiv Detail & Related papers (2026-08-05T14:28:05Z)
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
arXiv Detail & Related papers (2026-04-06T17:59:56Z)
GMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining [64.72014392166625]
GMS-CAVP is a novel framework that combines Multi-Scale Video-Audio Alignment and Multi-Scale Spatial-Temporal Diffusion-based pretraining objectives.<n>First, GMS-CAVP introduces a multi-scale contrastive learning strategy that captures semantic and temporal relations across varying granularities.<n>Second, we go beyond traditional contrastive learning by incorporating a diffusion-based generative objective, enabling modality translation and synthesis between video and audio.
arXiv Detail & Related papers (2026-01-27T13:43:32Z)
T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation [41.03487954415606]
Text-to-Audio-Video (T2AV) generation aims to synthesize temporally coherent video and semantically synchronized audio from natural language.<n>We present T2AV-, a unified benchmark for comprehensive evaluation of T2AV systems.<n>Even the strongest models fall substantially short of human-level realism and cross-modal consistency.
arXiv Detail & Related papers (2025-12-24T10:30:35Z)
Diff-V2M: A Hierarchical Conditional Diffusion Model with Explicit Rhythmic Modeling for Video-to-Music Generation [26.273309051211204]
Video-to-music (V2M) generation aims to create music that aligns with visual content.<n>We propose Diff-V2M, a general V2M framework based on a hierarchical conditional diffusion model.<n>For rhythm modeling, we begin by evaluating several rhythmic representations, including low-resolution mel-spectrograms, tempograms, and onset detection functions (ODF)
arXiv Detail & Related papers (2025-11-12T08:02:06Z)
StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing [63.72095377128904]
The visual dubbing task aims to generate mouth movements synchronized with the driving audio.<n>Audio-only driving paradigms inadequately capture speaker-specific lip habits.<n>Blind-inpainting approaches produce visual artifacts when handling obstructions.
arXiv Detail & Related papers (2025-09-26T05:23:31Z)
Controllable Video-to-Music Generation with Multiple Time-Varying Conditions [14.646682524511682]
Music enhances video narratives and emotions, driving demand for automatic video-to-music (V2M) generation.<n>Existing V2M methods relying solely on visual features or supplementary textual inputs generate music in a black-box manner.<n>We propose a novel multi-condition guided V2M generation framework that incorporates multiple time-varying conditions for enhanced control over music generation.
arXiv Detail & Related papers (2025-07-28T08:41:20Z)
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
arXiv Detail & Related papers (2025-06-29T06:41:00Z)
CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling [21.380988939240844]
We introduce a multi-modal diffusion model tailored for the bi-directional conditional generation of video and audio. We propose a joint contrastive training loss to improve the synchronization between visual and auditory occurrences.
arXiv Detail & Related papers (2023-12-08T23:55:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.