PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination

• URL: http://arxiv.org/abs/2604.12856v2
• Date: Wed, 15 Apr 2026 03:58:33 GMT
• Title: PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination
• Abstract summary: Bimanual piano motion generation requires precise modeling of complex musical structures and dynamic cross-hand coordination.<n>We propose PianoFlow, a flow-matching framework for precise and coordinated bimanual piano motion synthesis.<n>Our approach strategically leverages MIDI as a privileged modality during training, distilling these structured musical priors to achieve deep semantic understanding.
• Score: 31.224933689911186
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Audio-driven bimanual piano motion generation requires precise modeling of complex musical structures and dynamic cross-hand coordination. However, existing methods often rely on acoustic-only representations lacking symbolic priors, employ inflexible interaction mechanisms, and are limited to computationally expensive short-sequence generation. To address these limitations, we propose PianoFlow, a flow-matching framework for precise and coordinated bimanual piano motion synthesis. Our approach strategically leverages MIDI as a privileged modality during training, distilling these structured musical priors to achieve deep semantic understanding while maintaining audio-only inference. Furthermore, we introduce an asymmetric role-gated interaction module to explicitly capture dynamic cross-hand coordination through role-aware attention and temporal gating. To enable real-time streaming generation for arbitrarily long sequences, we design an autoregressive flow continuation scheme that ensures seamless cross-chunk temporal coherence. Extensive experiments on the PianoMotion10M dataset demonstrate that PianoFlow achieves superior quantitative and qualitative performance, while accelerating inference by over 9\times compared to previous methods.

Related papers

• Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation [17.906423604817395]
  We propose a novel hierarchical framework for minute-scale coherent music-to-dance generation.<n>Our method decouples the process into global planning and local temporal refinement, leveraging full-track musical context to ensure long-range coherence.<n>Our framework surpasses the conventional duration barrier, generating stable, 720p/30fps videos exceeding one minute with superior temporal stability.
  arXiv  Detail & Related papers  (2026-07-10T16:30:29Z)
• Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation [1.3115273020229667]
  Interactive music relies on real-time human expression.<n>Modern generative music AI is largely absent from this domain.<n>We propose a framework that bridges this gap.
  arXiv  Detail & Related papers  (2026-06-23T08:37:15Z)
• LK Jam: System Architecture and Implementation of a Real-Time Human-AI Interactive Music Generation System using Role-Aware GRU [5.407906676682284]
  LK_Jam is a real-time, bidirectional human-computer interactive music generation system.<n>It is based on a lightweight Gated Recurrent Unit (GRU) and a high-performance audio host architecture.
  arXiv  Detail & Related papers  (2026-06-19T01:03:44Z)
• MultiAct: Text-to-Motion Generation from Composite Text via Tailored Attention Guidance [54.131431576597514]
  MultiAct is an unpaired, inference-time framework for compositional text-to-motion synthesis.<n>Our method counteracts semantic collapse by adaptively amplifying cross-attention scores associated with underrepresented prompt components.
  arXiv  Detail & Related papers  (2026-05-29T07:14:48Z)
• ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning [32.581163427518185]
  ActionPlan bridges real-time streaming with high-quality offline generation within a single model.<n>Our real-time streaming is 5.25x faster while also achieving 18% motion quality improvement over the best previous method.
  arXiv  Detail & Related papers  (2026-03-13T18:24:32Z)
• Towards Arbitrary Motion Completing via Hierarchical Continuous Representation [64.6525112550758]
  We propose a novel parametric activation-induced hierarchical implicit representation framework, called NAME, based on Implicit Representations (INRs)<n>Our method introduces a hierarchical temporal encoding mechanism that extracts features from motion sequences at multiple temporal scales, enabling effective capture of intricate temporal patterns.
  arXiv  Detail & Related papers  (2025-12-24T14:07:04Z)
• YingMusic-Singer: Zero-shot Singing Voice Synthesis and Editing with Annotation-free Melody Guidance [16.462715982402884]
  Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment.<n>We propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody.<n>Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module.
  arXiv  Detail & Related papers  (2025-12-04T13:25:33Z)
• Planning with Sketch-Guided Verification for Physics-Aware Video Generation [71.29706409814324]
  We propose SketchVerify as a training-free, sketch-verification-based planning framework for video generation.<n>Our method predicts multiple candidate motion plans and ranks them using a vision-language verifier.<n>We iteratively refine the motion plan until a satisfactory one is identified, which is then passed to the trajectory-conditioned generator for final synthesis.
  arXiv  Detail & Related papers  (2025-11-21T17:48:02Z)
• Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation [62.14692332209628]
  "Interaction Distillation" is a novel training framework for more adequate preference modeling through attention-level optimization.<n>It provides more stable and generalizable reward signals compared to state-of-the-art RM optimization methods.
  arXiv  Detail & Related papers  (2025-08-04T17:06:23Z)
• Scaling Self-Supervised Representation Learning for Symbolic Piano Performance [52.661197827466886]
  We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions.<n>We use a comparatively smaller, high-quality subset to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings.
  arXiv  Detail & Related papers  (2025-06-30T14:00:14Z)
• MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation [10.203209816178552]
  MotionRAG-Diff is a hybrid framework that integrates Retrieval-Augmented Generation and diffusion-based refinement.<n>Our method introduces three core innovations.<n>It achieves state-of-the-art performance in motion quality, diversity, and music-motion synchronization accuracy.
  arXiv  Detail & Related papers  (2025-06-03T09:12:48Z)
• Extending Visual Dynamics for Video-to-Music Generation [51.274561293909926]
  DyViM is a novel framework to enhance dynamics modeling for video-to-music generation.<n>High-level semantics are conveyed through a cross-attention mechanism.<n>Experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-04-10T09:47:26Z)
• PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playing [7.026712993513959]
  We present PANDORA, a novel diffusion-based policy learning framework designed specifically for dexterous robotic piano performance.<n>Our approach employs a conditional U-Net architecture enhanced with FiLM-based global conditioning, which iteratively denoises noisy action sequences into smooth, high-dimensional trajectories.<n>We design a composite reward function that integrates task-specific accuracy, audio fidelity, and high-level semantic feedback from a large language model (LLM) oracle.
  arXiv  Detail & Related papers  (2025-03-17T17:22:34Z)
• Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization [19.27890803128116]
  We present a unified framework for automatic multitrack music arrangement.<n>At its core is a segment-level reconstruction objective operating on token-level disentangled content and style.<n>To support track-wise modeling, we introduce REMI-z, a structured tokenization scheme for multitrack symbolic music.
  arXiv  Detail & Related papers  (2024-08-27T16:18:51Z)
• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.