Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors

• URL: http://arxiv.org/abs/2604.09692v1
• Date: Mon, 06 Apr 2026 15:04:23 GMT
• Title: Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors
• Abstract summary: Piano motion exhibits a natural hierarchy: fingertip positions are nearly deterministic given piano geometry and fingering, while wrist and intermediate joints offer stylistic freedom.<n>We present [OURS], a four-stage framework exploiting this hierarchy, including statistics-based fingertip positioning, FiLM-conditioned trajectory refinement, and wrist estimation.
• Score: 13.063302264614057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthesizing realistic piano hand motions requires both precision and naturalness. Physics-based methods achieve precision but produce stiff motions; data-driven models learn natural dynamics but struggle with positional accuracy. Piano motion exhibits a natural hierarchy: fingertip positions are nearly deterministic given piano geometry and fingering, while wrist and intermediate joints offer stylistic freedom. We present [OURS], a four-stage framework exploiting this hierarchy: (1) statistics-based fingertip positioning, (2) FiLM-conditioned trajectory refinement, (3) wrist estimation, and (4) STGCN-based pose synthesis. We contribute expert-annotated fingerings for the FürElise dataset (153 pieces, ~10 hours). Experiments demonstrate F1 = 0.910, substantially outperforming diffusion baselines (F1 = 0.121), with user study (N=41) confirming quality approaching motion capture. Expert evaluation by professional pianists (N=5) identified anticipatory motion as the key remaining gap, providing concrete directions for future improvement.

Related papers

• MUSIC: Learning Muscle-Driven Dexterous Hand Control [22.43764880740859]
  We present a data-driven approach for physics-based, muscle-driven dexterous control.<n>Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination.<n>We show that our approach can synthesize coordinated bimanual motions with accurate key presses.
  arXiv  Detail & Related papers  (2026-04-26T21:05:34Z)
• HandX: Scaling Bimanual Motion and Interaction Generation [55.486397101415]
  HandX is a unified foundation spanning data, annotation, and evaluation.<n>We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset.<n>Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics.
  arXiv  Detail & Related papers  (2026-03-30T17:59:49Z)
• PHANTOM Hand [12.871356369910323]
  Tendon-driven underactuated hands excel in adaptive grasping but often suffer from kinematic unpredictability and highly non-actuated force transmission.<n>We introduce the PHANTOM Hand (Hybrid Precision-Augmented Compliance): a modular, human-scale system featuring 6 actuators and 15 degrees of freedom (DoFs)<n>We propose a unified framework that bridges the gap between precise analytic shaping and robust compliant grasping.
  arXiv  Detail & Related papers  (2026-03-24T12:52:18Z)
• HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policies [32.80504141722999]
  HandelBot is a framework that combines a simulation policy and rapid adaptation through a two-stage pipeline.<n>We demonstrate that HandelBot can successfully perform precise bimanual piano playing.<n>Our system outperforms direct simulation deployment by a factor of 1.8x and requires only 30 minutes of physical interaction data.
  arXiv  Detail & Related papers  (2026-03-12T17:56:29Z)
• PianoVAM: A Multimodal Piano Performance Dataset [56.318475235705954]
  PianoVAM is a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata.<n>The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions.<n>Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm.
  arXiv  Detail & Related papers  (2025-09-10T17:35:58Z)
• Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation [88.83749146867665]
  Existing approaches learn a policy to predict a distant next-best end-effector pose.<n>They then compute the corresponding joint rotation angles for motion using inverse kinematics.<n>We propose Kinematics enhanced Spatial-TemporAl gRaph diffuser.
  arXiv  Detail & Related papers  (2025-03-13T17:48:35Z)
• FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance [15.909113091360206]
  Hand motion models with the sophistication to accurately recreate piano playing have a wide range of applications in character animation, embodied AI, biomechanics, and VR/AR. In this paper, we construct a first-of-its-kind large-scale dataset that contains approximately 10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153 pieces of classical music.
  arXiv  Detail & Related papers  (2024-10-08T08:21:05Z)
• RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands [57.64308229980045]
  We introduce the Robot Piano 1 Million dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.
  arXiv  Detail & Related papers  (2024-08-20T17:56:52Z)
• PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance [15.21347897534943]
  We construct a piano-hand motion generation benchmark to guide hand movements and fingerings for piano playing.<n>To this end, we collect an annotated dataset, PianoMotion10M, consisting of 116 hours of piano playing videos from a bird's-eye view with 10 million annotated hand poses.
  arXiv  Detail & Related papers  (2024-06-13T17:05:23Z)
• A Skeleton-Driven Neural Occupancy Representation for Articulated Hands [49.956892429789775]
  Hand ArticuLated Occupancy (HALO) is a novel representation of articulated hands that bridges the advantages of 3D keypoints and neural implicit surfaces. We demonstrate the applicability of HALO to the task of conditional generation of hands that grasp 3D objects.
  arXiv  Detail & Related papers  (2021-09-23T14:35:19Z)
• Towards Learning to Play Piano with Dexterous Hands and Touch [79.48656721563795]
  We demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano. We achieve this by using a touch-augmented reward and a novel curriculum of tasks.
  arXiv  Detail & Related papers  (2021-06-03T17:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.