TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation

• URL: http://arxiv.org/abs/2605.01809v1
• Date: Sun, 03 May 2026 10:25:47 GMT
• Title: TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation
• Abstract summary: TMD-Bench is a benchmark for text-driven music-dance co-generation.<n>It assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment.
• Score: 45.3416170884362
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Unified audio-visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive media. However, when moving from general audio-video synthesis to music-dance co-generation, the task becomes substantially harder: musical rhythm, phrasing, and accents must drive choreographic motion at fine temporal resolution, and such rhythmic coupling is not captured by unimodal metrics or generic audiovisual consistency scores used in current evaluation practice. We introduce TMD-Bench, a benchmark for text-driven music-dance co-generation that assesses systems across unimodal generation quality, instruction adherence, and cross-modal rhythmic alignment. The benchmark integrates computable physical metrics with perceptual multimodal judgments, and is supported by a curated rhythm-aligned music-dance dataset and a fine-grained Music Captioner for structured music semantics. TMD-Bench further reveals that (i) modern commercial audio-visual models, such as Veo 3 and Sora 2, produce high-quality music and video, while rhythmic coupling remains less consistently optimized and leaves room for improvement, and (ii) our unified baseline RhyJAM trained on rhythm-aligned data achieves competitive beat-level synchronization while maintaining competitive unimodal fidelity. This presents prospects for building next-generation music-dance models that explicitly optimize rhythmic and kinetic coherence.

Related papers

• CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation [36.54868745472302]
  Dance-to-music generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos.<n>Existing methods typically rely on sparse cues and supervise only the final audio output.<n>We propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport.
  arXiv  Detail & Related papers  (2026-09-11T17:46:15Z)
• VIBE: Video Instruction-aligned Background music gEneration [58.326977744404566]
  Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations.<n>We introduce VIBE, a novel text-and-video-to-music (T+V2M) generation model that leverages Conditioning Connection.
  arXiv  Detail & Related papers  (2026-08-31T01:19:55Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• Listen to Rhythm, Choose Movements: Autoregressive Multimodal Dance Generation via Diffusion and Mamba with Decoupled Dance Dataset [8.721362823189077]
  Listen to Rhythm, Choose Movements (LRCM) is a multimodal-guided diffusion framework supporting both diverse input modalities and autoregressive dance motion generation.<n>We will release the full dataset, and pretrained models publicly upon acceptance.
  arXiv  Detail & Related papers  (2026-01-06T14:59:22Z)
• Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation [62.82943523102]
  Music to 3D dance generation aims to synthesize realistic and rhythmically synchronized human dance from music.<n>We propose TempoMoE, a hierarchical tempo-aware Mixture-of-Experts module.<n>We show that TempoMoE achieves state-of-the-art results in dance quality and rhythm alignment.
  arXiv  Detail & Related papers  (2025-12-21T16:57:08Z)
• GACA-DiT: Diffusion-based Dance-to-Music Generation with Genre-Adaptive Rhythm and Context-Aware Alignment [16.93446224499017]
  Dance-to-music (D2M) generation aims to automatically compose music that is rhythmically and temporally aligned with dance movements.<n>We propose textbfGACA-DiT, a diffusion transformer-based framework with two novel modules for rhythmically consistent and temporally aligned music generation.<n>Experiments on the AIST++ and TikTok datasets demonstrate that GACA-DiT outperforms state-of-the-art methods in both objective metrics and human evaluation.
  arXiv  Detail & Related papers  (2025-10-28T09:26:59Z)
• MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding [13.25040795516169]
  MotionBeat is a framework for motion-aligned music representation learning.<n>We show that MotionBeat outperforms state-of-the-art audio encoders in music-to-dance generation.
  arXiv  Detail & Related papers  (2025-10-15T07:44:32Z)
• Let Your Video Listen to Your Music! [62.27731415767459]
  We propose a novel framework, MVAA, that automatically edits video to align with the rhythm of a given music track.<n>We modularize the task into a two-step process in our MVAA: aligning motion with audio beats, followed by rhythm-aware video editing.<n>This hybrid approach enables adaptation within 10 minutes with one on a single NVIDIA 4090 GPU using CogVideoX-5b-I2V as the backbone.
  arXiv  Detail & Related papers  (2025-06-23T17:52:16Z)
• MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation [10.203209816178552]
  MotionRAG-Diff is a hybrid framework that integrates Retrieval-Augmented Generation and diffusion-based refinement.<n>Our method introduces three core innovations.<n>It achieves state-of-the-art performance in motion quality, diversity, and music-motion synchronization accuracy.
  arXiv  Detail & Related papers  (2025-06-03T09:12:48Z)
• Extending Visual Dynamics for Video-to-Music Generation [51.274561293909926]
  DyViM is a novel framework to enhance dynamics modeling for video-to-music generation.<n>High-level semantics are conveyed through a cross-attention mechanism.<n>Experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-04-10T09:47:26Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.