Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

• URL: http://arxiv.org/abs/2606.22726v1
• Date: Mon, 22 Jun 2026 00:01:05 GMT
• Title: Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation
• Abstract summary: STREAM is a modality-decoupled diffusion transformer for Choreographic Motion Generation.<n>We introduce Motorica++, a dataset enriched with domain-specific dance vocabulary and frame-level semantic annotations.<n>STREAM achieves state-of-the-art alignment between motion and music while perfectly preserving choreographic semantics.
• Score: 11.963078031832088
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics. While existing models can synthesize motion from music, they remain largely black boxes. Conversely, attempting to condition generation on both text and music frequently leads to modality collapse, where dense acoustic rhythms overwhelm sparse semantic text prompts, destroying user controllability. To resolve this spatial-temporal conflict, we propose STREAM (Structural-Temporal Rhythmic Energy-based Attention for Motion), a modality-decoupled diffusion transformer. STREAM strictly separates conditioning pathways: global text semantics dictate the kinematic structure via Adaptive Layer Normalization (AdaLN), while a novel Bimodal Energy-Based Attention Module (BEAM) routes these features to the musical beat without overwriting the semantics. We further introduce Motorica++, a newly curated dataset enriched with domain-specific dance vocabulary and frame-level semantic annotations from existing Motorica dataset. Additionally, to rigorously quantify zero-shot editability, we propose the Exchange Evaluation Protocol and Editable Dance Score (EDS). Through extensive experiments, STREAM achieves state-of-the-art alignment between motion and music while perfectly preserving choreographic semantics, positioning AI not merely as a reactive synthesizer, but as a controllable, collaborative partner for artistic direction. The source code and datasets are available at https://github.com/SeongJong-Yoo/STREAM.

Related papers

• Spatial Temporal Synergy: Balancing Change and Invariance in Text Driven 3D Human Motion Editing [78.44034844227868]
  Text-driven human motion editing aims to modify existing motion sequences according to natural language instructions.<n>Change and Invariance Motion Editing (CIME) is a unified framework that comprehensively decouples change and invariance into spatial pose and temporal rhythm dimensions.<n>Experiments on the MotionFix and STANCE Adjustment datasets demonstrate that CIME achieves state-of-the-art performance in editing alignment and structural fidelity.
  arXiv  Detail & Related papers  (2026-08-17T01:54:44Z)
• Music-to-Dance Generation via Atomic Movements [67.09789936557199]
  Music-driven dance generation aims to produce human motion that is rhythmically synchronized and semantically consistent with music.<n>We introduce a structure-aware framework that models choreography as a sequence of atomic movements-semantically interpretable motion events.<n>Our method produces dances with significantly improved structural coherence, rhythmic alignment, and perceptual naturalness compared to existing baselines.
  arXiv  Detail & Related papers  (2026-07-15T16:03:55Z)
• TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation [32.964897427586166]
  TeMuDance is a framework that enables text-based control for music-conditioned dance generation.<n>It uses motion as a shared semantic anchor to align disjoint music-dance and text-motion datasets.<n>It achieves competitive dance quality while substantially improving text-conditioned control over existing methods.
  arXiv  Detail & Related papers  (2026-04-18T14:24:57Z)
• Listen to Rhythm, Choose Movements: Autoregressive Multimodal Dance Generation via Diffusion and Mamba with Decoupled Dance Dataset [8.721362823189077]
  Listen to Rhythm, Choose Movements (LRCM) is a multimodal-guided diffusion framework supporting both diverse input modalities and autoregressive dance motion generation.<n>We will release the full dataset, and pretrained models publicly upon acceptance.
  arXiv  Detail & Related papers  (2026-01-06T14:59:22Z)
• GACA-DiT: Diffusion-based Dance-to-Music Generation with Genre-Adaptive Rhythm and Context-Aware Alignment [16.93446224499017]
  Dance-to-music (D2M) generation aims to automatically compose music that is rhythmically and temporally aligned with dance movements.<n>We propose textbfGACA-DiT, a diffusion transformer-based framework with two novel modules for rhythmically consistent and temporally aligned music generation.<n>Experiments on the AIST++ and TikTok datasets demonstrate that GACA-DiT outperforms state-of-the-art methods in both objective metrics and human evaluation.
  arXiv  Detail & Related papers  (2025-10-28T09:26:59Z)
• MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding [13.25040795516169]
  MotionBeat is a framework for motion-aligned music representation learning.<n>We show that MotionBeat outperforms state-of-the-art audio encoders in music-to-dance generation.
  arXiv  Detail & Related papers  (2025-10-15T07:44:32Z)
• DanceEditor: Towards Iterative Editable Music-driven Dance Generation with Open-Vocabulary Descriptions [22.816939666772303]
  We propose a novel framework for iterative and editable dance generation coherently aligned with given music signals.<n>Our framework is built upon a prediction-then-editing paradigm unifying multi-modal conditions.<n>Our method outperforms the state-of-the-art models on our newly collected DanceRemix dataset.
  arXiv  Detail & Related papers  (2025-08-24T12:53:09Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• UniMuMo: Unified Text, Music and Motion Generation [57.72514622935806]
  We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture.
  arXiv  Detail & Related papers  (2024-10-06T16:04:05Z)
• TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration [75.37311932218773]
  We propose a novel task for generating 3D dance movements that simultaneously incorporate both text and music modalities. Our approach can generate realistic and coherent dance movements conditioned on both text and music while maintaining comparable performance with the two single modalities.
  arXiv  Detail & Related papers  (2023-04-05T12:58:33Z)
• BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis [123.73677487809418]
  We introduce a new dataset aiming to challenge common assumptions in dance motion synthesis. We focus on breakdancing which features acrobatic moves and tangled postures. Our efforts produced the BRACE dataset, which contains over 3 hours and 30 minutes of densely annotated poses.
  arXiv  Detail & Related papers  (2022-07-20T18:03:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.