From Score to Sound: An End-to-End MIDI-to-Motion Pipeline for Robotic Cello Performance

• URL: http://arxiv.org/abs/2601.03562v1
• Date: Wed, 07 Jan 2026 04:11:00 GMT
• Title: From Score to Sound: An End-to-End MIDI-to-Motion Pipeline for Robotic Cello Performance
• Abstract summary: We propose a novel end-to-end MIDI score to robotic motion pipeline.<n>This pipeline converts musical input directly into collision-aware bowing motions for a UR5e robot cellist.<n>To demonstrate the effectiveness of our method in comparison to human performers, we introduce the Musical Turing Test.
• Score: 1.6971047370270889
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robot musicians require precise control to obtain proper note accuracy, sound quality, and musical expression. Performance of string instruments, such as violin and cello, presents a significant challenge due to the precise control required over bow angle and pressure to produce the desired sound. While prior robotic cellists focus on accurate bowing trajectories, these works often rely on expensive motion capture techniques, and fail to sightread music in a human-like way. We propose a novel end-to-end MIDI score to robotic motion pipeline which converts musical input directly into collision-aware bowing motions for a UR5e robot cellist. Through use of Universal Robot Freedrive feature, our robotic musician can achieve human-like sound without the need for motion capture. Additionally, this work records live joint data via Real-Time Data Exchange (RTDE) as the robot plays, providing labeled robotic playing data from a collection of five standard pieces to the research community. To demonstrate the effectiveness of our method in comparison to human performers, we introduce the Musical Turing Test, in which a collection of 132 human participants evaluate our robot's performance against a human baseline. Human reference recordings are also released, enabling direct comparison for future studies. This evaluation technique establishes the first benchmark for robotic cello performance. Finally, we outline a residual reinforcement learning methodology to improve upon baseline robotic controls, highlighting future opportunities for improved string-crossing efficiency and sound quality.

Related papers

• Expressive Robotic Pianist: Mastering Complex Piano Repertoire with Graph-Mimic and Musical Dynamics [5.506290956393904]
  We show that a dexterous robotic hand can achieve high-fidelity performance across a diverse piano repertoire.<n>Our approach provides a robust pathway for robotic systems to move beyond mere mechanical accuracy.
  arXiv  Detail & Related papers  (2026-09-09T21:26:29Z)
• Humanoid Musical Robots as Experimental Interfaces for Music-Evoked Emotion [1.3750624267664158]
  We argue that musical humanoid robots are experimental interfaces for investigating music-evoked emotions.<n>We show that humanoid robots are well-suited as they enable parametric control of performance variables.<n>Our work positions musical humanoid robots as a methodological platform that enables future controlled investigations of music-evoked emotions.
  arXiv  Detail & Related papers  (2026-08-20T03:23:36Z)
• A Haptic Robot Finger Designed for Guqin Instrument Playing [9.847859242319275]
  Musical instrument-playing robots have emerged in recent years, such as pianos, chime bells, and taiko drums.<n>These robots primarily employ open-loop positional control, rendering them incapable of operating instruments requiring dexterous hands.<n>This paper describes the design and validation of a high-precision tactile-sensing finger.
  arXiv  Detail & Related papers  (2026-08-07T09:18:58Z)
• HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos [58.9564236347451]
  HumanEgo is a framework that bridges the embodiment gap between human and robot.<n>It lifts each human demonstration to an entity-level representation of hand-object interaction.<n>HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable.
  arXiv  Detail & Related papers  (2026-05-24T08:26:41Z)
• Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control [52.83779852397341]
  Current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands.<n>We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio.<n>RoboPerform achieves a ResMoE policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection.
  arXiv  Detail & Related papers  (2025-12-29T17:59:24Z)
• H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [58.006918399913665]
  We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos.<n>Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale.<n>At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions.
  arXiv  Detail & Related papers  (2025-12-10T07:59:45Z)
• Design and Control of a Bipedal Robotic Character [3.650193138379926]
  This work aims to unify expressive, artist-directed motions and robust dynamic mobility for legged robots.<n>We introduce a new bipedal robot, designed with a focus on character-driven mechanical features.<n>We present a reinforcement learning-based control architecture to robustly execute artistic motions conditioned on command signals.
  arXiv  Detail & Related papers  (2025-01-09T12:55:21Z)
• Breathless: An 8-hour Performance Contrasting Human and Robot Expressiveness [16.69221972067975]
  This paper describes the robot technology behind an original performance that pairs a human dancer with an industrial robot arm.<n>To control the robot arm, we combine a range of sinusoidal motions with varying amplitude, frequency and offset at each joint to evoke human motions common in physical labor.<n>More motions were developed using deep learning techniques for video-based human-pose tracking and extraction.
  arXiv  Detail & Related papers  (2024-11-19T09:20:51Z)
• RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands [57.64308229980045]
  We introduce the Robot Piano 1 Million dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.
  arXiv  Detail & Related papers  (2024-08-20T17:56:52Z)
• Improving Drumming Robot Via Attention Transformer Network [2.5064819128982556]
  We introduce an improving drumming robot that can automatically complete music transcription based on the popular vision transformer network based on the attention mechanism. Massive experimental results demonstrate that the improving algorithm can help the drumming robot promote drum classification performance.
  arXiv  Detail & Related papers  (2023-10-04T03:55:43Z)
• ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space [9.806227900768926]
  This paper introduces a novel deep-learning approach for human-to-robot motion. Our method does not require paired human-to-robot data, which facilitates its translation to new robots. Our model outperforms existing works regarding human-to-robot similarity in terms of efficiency and precision.
  arXiv  Detail & Related papers  (2023-09-11T08:55:04Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning [61.10744686260994]
  We introduce RoboPianist, a system that enables simulated anthropomorphic hands to learn an extensive repertoire of 150 piano pieces. We additionally introduce an open-sourced environment, benchmark of tasks, interpretable evaluation metrics, and open challenges for future study.
  arXiv  Detail & Related papers  (2023-04-09T03:53:05Z)
• HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration [57.045140028275036]
  We show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning. We propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy.
  arXiv  Detail & Related papers  (2022-12-08T15:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.