HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization

• URL: http://arxiv.org/abs/2503.01725v2
• Date: Tue, 04 Mar 2025 15:31:11 GMT
• Title: HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronization
• Abstract summary: This paper introduces HarmonySet, a comprehensive dataset designed to advance video-music understanding.<n> HarmonySet consists of 48,328 diverse video-music pairs, annotated with detailed information on rhythmic synchronization, emotional alignment, thematic coherence, and cultural relevance.
• Score: 11.659199378890218
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces HarmonySet, a comprehensive dataset designed to advance video-music understanding. HarmonySet consists of 48,328 diverse video-music pairs, annotated with detailed information on rhythmic synchronization, emotional alignment, thematic coherence, and cultural relevance. We propose a multi-step human-machine collaborative framework for efficient annotation, combining human insights with machine-generated descriptions to identify key transitions and assess alignment across multiple dimensions. Additionally, we introduce a novel evaluation framework with tasks and metrics to assess the multi-dimensional alignment of video and music, including rhythm, emotion, theme, and cultural context. Our extensive experiments demonstrate that HarmonySet, along with the proposed evaluation framework, significantly improves the ability of multimodal models to capture and analyze the intricate relationships between video and music.

Related papers

• The MPB Corpus: A Dataset of Melody, Rhythm, Harmony, and Melody-Harmony Relationships in Brazilian Popular Music [0.0]
  The MPB Corpus is a collection of 500 musical pieces encoded across four musical parameters.<n>It constitutes the most comprehensive and detailed dataset to date for computational musicology of Brazilian music.
  arXiv  Detail & Related papers  (2026-08-14T00:17:10Z)
• TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation [45.3416170884362]
  TMD-Bench is a benchmark for text-driven music-dance co-generation.<n>It assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment.
  arXiv  Detail & Related papers  (2026-05-03T10:25:47Z)
• A Study on the Data Distribution Gap in Music Emotion Recognition [7.281487567929003]
  Music Emotion Recognition (MER) is a task deeply connected to human perception.<n>Prior studies tend to focus on specific musical styles rather than incorporating a diverse range of genres.<n>We address the task of recognizing emotion from audio content by investigating five datasets with dimensional emotion annotations.
  arXiv  Detail & Related papers  (2025-10-06T10:57:05Z)
• EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation [5.834095167258525]
  We propose a generative model leveraging multimodal visual controls - specifically, human facial expressions and upper-body motion.<n>We adopt parameter-efficient fine-tuning (PEFT) on the pretrained text-to-music generation model.<n>We introduce a novel dataset consisting of 7 hours of synchronized video recordings capturing expressive facial and upper-body gestures aligned with corresponding music.
  arXiv  Detail & Related papers  (2025-07-07T12:56:20Z)
• TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure [8.721294663967305]
  We introduce TOMI (Transforming and Organizing Music Ideas) as a novel approach in deep music generation.<n>We represent a multi-track composition process via a sparse, four-dimensional space characterized by clips (short audio or MIDI segments), sections (temporal positions), tracks (instrument layers) and transformations.<n>Our model is capable of generating multi-track electronic music with full-song structure, and we further integrate the TOMI-based model with the REAPER digital audio workstation.
  arXiv  Detail & Related papers  (2025-06-29T05:15:41Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [52.33281620699459]
  ThinkSound is a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning [20.072410431003142]
  DeepResonance is a multimodal music understanding model fine-tuned for multi-way aligned music, text, image, and video data.<n>We construct datasets designed to enable DeepResonance to integrate both visual and textual music feature content.<n>Our model achieves state-of-the-art performances across six music understanding tasks.
  arXiv  Detail & Related papers  (2025-02-18T08:09:42Z)
• MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization [52.498942604622165]
  This paper presents MuVi, a framework to generate music that aligns with video content. MuVi analyzes video content through a specially designed visual adaptor to extract contextually and temporally relevant features. We show that MuVi demonstrates superior performance in both audio quality and temporal synchronization.
  arXiv  Detail & Related papers  (2024-10-16T18:44:56Z)
• UniMuMo: Unified Text, Music and Motion Generation [57.72514622935806]
  We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture.
  arXiv  Detail & Related papers  (2024-10-06T16:04:05Z)
• Unlocking Potential in Pre-Trained Music Language Models for Versatile Multi-Track Music Arrangement [10.714947060480426]
  We propose a unified sequence-to-sequence framework that enables the fine-tuning of a symbolic music language model. Our experiments demonstrate that the proposed approach consistently achieves higher musical quality compared to task-specific baselines.
  arXiv  Detail & Related papers  (2024-08-27T16:18:51Z)
• MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions [69.9122231800796]
  We present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions.<n>We propose a systemic captioning framework, achieving various modality annotations with more than 27.1k hours of trailer videos.<n>Our dataset potentially paves the path for fine-grained large multimodal-language model training.
  arXiv  Detail & Related papers  (2024-07-30T16:43:24Z)
• MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models [57.47799823804519]
  We are inspired by how musicians compose music not just from a movie script, but also through visualizations. We propose MeLFusion, a model that can effectively use cues from a textual description and the corresponding image to synthesize music. Our exhaustive experimental evaluation suggests that adding visual information to the music synthesis pipeline significantly improves the quality of generated music.
  arXiv  Detail & Related papers  (2024-06-07T06:38:59Z)
• Synchformer: Efficient Synchronization from Sparse Cues [100.89656994681934]
  Our contributions include a novel audio-visual synchronization model, and training that decouples extraction from synchronization modelling. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.
  arXiv  Detail & Related papers  (2024-01-29T18:59:55Z)
• And what if two musical versions don't share melody, harmony, rhythm, or lyrics ? [2.4366811507669124]
  We show that an approximated representation of the lyrics is an efficient proxy to discriminate between versions and non-versions. We then describe how these features complement each other and yield new state-of-the-art performances on two publicly available datasets.
  arXiv  Detail & Related papers  (2022-10-03T22:33:14Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Deep Music Information Dynamics [1.6143012623830792]
  We introduce a novel framework that combines two parallel streams - a low rate latent representation stream and a higher rate information dynamics derived from the musical data itself. Motivated by rate-distortion theories of human cognition we propose a framework for exploring possible relations between imaginary anticipations existing in the listener's mind and information dynamics of the musical surface itself.
  arXiv  Detail & Related papers  (2021-02-01T19:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.