Learning to Play Piano in the Real World

• URL: http://arxiv.org/abs/2503.15481v1
• Date: Wed, 19 Mar 2025 17:56:14 GMT
• Title: Learning to Play Piano in the Real World
• Authors: Yves-Simon Zeulner, Sandeep Selvaraj, Roberto Calandra,
• Abstract summary: We develop the first piano playing robotic system that makes use of learning approaches while also being deployed on a real world dexterous robot.<n>Specifically, we make use of Sim2Real to train a policy in simulation using reinforcement learning before deploying the learned policy on a real world dexterous robot.
• Score: 3.824631943614614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Towards the grand challenge of achieving human-level manipulation in robots, playing piano is a compelling testbed that requires strategic, precise, and flowing movements. Over the years, several works demonstrated hand-designed controllers on real world piano playing, while other works evaluated robot learning approaches on simulated piano scenarios. In this paper, we develop the first piano playing robotic system that makes use of learning approaches while also being deployed on a real world dexterous robot. Specifically, we make use of Sim2Real to train a policy in simulation using reinforcement learning before deploying the learned policy on a real world dexterous robot. In our experiments, we thoroughly evaluate the interplay between domain randomization and the accuracy of the dynamics model used in simulation. Moreover, we evaluate the robot's performance across multiple songs with varying complexity to study the generalization of our learned policy. By providing a proof-of-concept of learning to play piano in the real world, we want to encourage the community to adopt piano playing as a compelling benchmark towards human-level manipulation. We open-source our code and show additional videos at https://lasr.org/research/learning-to-play-piano .

Related papers

• Multi-Robot Collaboration through Reinforcement Learning and Abstract Simulation [3.5253513747455303]
  We investigate the extent to which so-called abstract simulators can be used for multi-agent reinforcement learning.<n>An abstract simulator models the robot's target task at a high-level of abstraction and discards many details of the world that could impact optimal decision-making.<n> Policies are trained in an abstract simulator then transferred to the physical robot by making use of separately-obtained low-level perception and motion control modules.
  arXiv  Detail & Related papers  (2025-03-07T02:23:24Z)
• RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands [57.64308229980045]
  We introduce the Robot Piano 1 Million dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.
  arXiv  Detail & Related papers  (2024-08-20T17:56:52Z)
• RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning [61.10744686260994]
  We introduce RoboPianist, a system that enables simulated anthropomorphic hands to learn an extensive repertoire of 150 piano pieces. We additionally introduce an open-sourced environment, benchmark of tasks, interpretable evaluation metrics, and open challenges for future study.
  arXiv  Detail & Related papers  (2023-04-09T03:53:05Z)
• A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning [86.06110576808824]
  Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments. Recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped in only 20 minutes in the real world.
  arXiv  Detail & Related papers  (2022-08-16T17:37:36Z)
• DayDreamer: World Models for Physical Robot Learning [142.11031132529524]
  Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn. Many advances in robot learning rely on simulators. In this paper, we apply Dreamer to 4 robots to learn online and directly in the real world, without simulators.
  arXiv  Detail & Related papers  (2022-06-28T17:44:48Z)
• Robot Learning from Randomized Simulations: A Review [59.992761565399185]
  Deep learning has caused a paradigm shift in robotics research, favoring methods that require large amounts of data. State-of-the-art approaches learn in simulation where data generation is fast as well as inexpensive. We focus on a technique named 'domain randomization' which is a method for learning from randomized simulations.
  arXiv  Detail & Related papers  (2021-11-01T13:55:41Z)
• Towards Learning to Play Piano with Dexterous Hands and Touch [79.48656721563795]
  We demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano. We achieve this by using a touch-augmented reward and a novel curriculum of tasks.
  arXiv  Detail & Related papers  (2021-06-03T17:59:31Z)
• High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewards [35.55280687116388]
  We show that a learning system can learn to juggle in the real-world without harming the robot. We demonstrate that this system enables the high-speed Barrett WAM manipulator to learn juggling two balls from 56 minutes of experience with a binary reward signal. The final policy juggles continuously for up to 33 minutes or about 4500 repeated catches.
  arXiv  Detail & Related papers  (2020-10-26T11:13:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.