Towards Learning to Play Piano with Dexterous Hands and Touch
- URL: http://arxiv.org/abs/2106.02040v1
- Date: Thu, 3 Jun 2021 17:59:31 GMT
- Title: Towards Learning to Play Piano with Dexterous Hands and Touch
- Authors: Huazhe Xu, Yuping Luo, Shaoxiong Wang, Trevor Darrell, Roberto
Calandra
- Abstract summary: We demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano.
We achieve this by using a touch-augmented reward and a novel curriculum of tasks.
- Score: 79.48656721563795
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The virtuoso plays the piano with passion, poetry and extraordinary technical
ability. As Liszt said (a virtuoso)must call up scent and blossom, and breathe
the breath of life. The strongest robots that can play a piano are based on a
combination of specialized robot hands/piano and hardcoded planning algorithms.
In contrast to that, in this paper, we demonstrate how an agent can learn
directly from machine-readable music score to play the piano with dexterous
hands on a simulated piano using reinforcement learning (RL) from scratch. We
demonstrate the RL agents can not only find the correct key position but also
deal with various rhythmic, volume and fingering, requirements. We achieve this
by using a touch-augmented reward and a novel curriculum of tasks. We conclude
by carefully studying the important aspects to enable such learning algorithms
and that can potentially shed light on future research in this direction.
Related papers
- PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrations [21.52466727496551]
We introduce PianoMime, a framework for training a piano-playing agent using internet demonstrations.
In our work, we leverage these demonstrations to learn a generalist piano-playing agent capable of playing any arbitrary song.
We show that we are able to learn a policy with up to 56% F1 score on unseen songs.
arXiv Detail & Related papers (2024-07-25T16:37:07Z) - PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Training [8.484581633133542]
PianoBART is a pre-trained model that uses BART for both symbolic piano music generation and understanding.
We devise a multi-level object selection strategy for different pre-training tasks of PianoBART, which can prevent information leakage or loss.
Experiments demonstrate that PianoBART efficiently learns musical patterns and achieves outstanding performance in generating high-quality coherent pieces.
arXiv Detail & Related papers (2024-06-26T03:35:54Z) - PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance [15.21347897534943]
We construct a piano-hand motion generation benchmark to guide hand movements and fingerings for piano playing.
To this end, we collect an annotated dataset, PianoMotion10M, consisting of 116 hours of piano playing videos from a bird's-eye view with 10 million annotated hand poses.
arXiv Detail & Related papers (2024-06-13T17:05:23Z) - RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning [61.10744686260994]
We introduce RoboPianist, a system that enables simulated anthropomorphic hands to learn an extensive repertoire of 150 piano pieces.
We additionally introduce an open-sourced environment, benchmark of tasks, interpretable evaluation metrics, and open challenges for future study.
arXiv Detail & Related papers (2023-04-09T03:53:05Z) - At Your Fingertips: Extracting Piano Fingering Instructions from Videos [45.643494669796866]
We consider the AI task of automating the extraction of fingering information from videos.
We show how to perform this task with high-accuracy using a combination of deep-learning modules.
We run the resulting system on 90 videos, resulting in high-quality piano fingering information of 150K notes.
arXiv Detail & Related papers (2023-03-07T09:09:13Z) - Optimizing piano practice with a utility-based scaffold [59.821144959060305]
A typical part of learning to play the piano is the progression through a series of practice units that focus on individual dimensions of the skill.
Because we each learn differently, and because there are many choices for possible piano practice tasks and methods, the set of practice tasks should be dynamically adapted to the human learner.
We present a modeling framework to guide the human learner through the learning process by choosing practice modes that have the highest expected utility.
arXiv Detail & Related papers (2021-06-21T14:05:00Z) - Dance Revolution: Long-Term Dance Generation with Music via Curriculum
Learning [55.854205371307884]
We formalize the music-conditioned dance generation as a sequence-to-sequence learning problem.
We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation.
Our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation.
arXiv Detail & Related papers (2020-06-11T00:08:25Z) - RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement
Learning [69.20460466735852]
This paper presents a deep reinforcement learning algorithm for online accompaniment generation.
The proposed algorithm is able to respond to the human part and generate a melodic, harmonic and diverse machine part.
arXiv Detail & Related papers (2020-02-08T03:53:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.