Pay Attention to the Keys: Visual Piano Transcription Using Transformers

• URL: http://arxiv.org/abs/2411.09037v2
• Date: Fri, 04 Jul 2025 14:46:39 GMT
• Title: Pay Attention to the Keys: Visual Piano Transcription Using Transformers
• Abstract summary: Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone.<n>We propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs)<n>We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone (e.g., from a top-down video of the piano keyboard). In this work we propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs). Our system is trained on the newly introduced R3 dataset, consisting of ca.~31 hours of synchronized video and MIDI recordings of piano performances. We additionally introduce an approach to predict note offsets, which has not been previously explored in this context. We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets.

Related papers

• Multi-Task Multi-Frame Visual Piano Transcription [57.32059378506694]
  We present V2N (Video to Notes), the first complete Visual Piano Transcription system.<n>A shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity.<n>Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy.
  arXiv  Detail & Related papers  (2026-08-04T10:11:45Z)
• D3PIA: A Discrete Denoising Diffusion Model for Piano Accompaniment Generation From Lead sheet [18.81856284996403]
  We propose a discrete diffusion-based piano accompaniment generation model, D3PIA, using local alignment between lead sheet and accompaniment in piano-roll representation.<n>We evaluate our model using the POP909 dataset, a widely used benchmark for piano accompaniment generation.
  arXiv  Detail & Related papers  (2026-02-03T13:39:33Z)
• PianoVAM: A Multimodal Piano Performance Dataset [56.318475235705954]
  PianoVAM is a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata.<n>The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions.<n>Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm.
  arXiv  Detail & Related papers  (2025-09-10T17:35:58Z)
• Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations [2.3249139042158853]
  We present a neural network approach for synchronizing audio recordings of human piano performances with their corresponding loosely aligned MIDI files.<n>The proposed model achieves up to 20% higher alignment accuracy than the industry-standard Dynamic Time Warping (DTW) method.
  arXiv  Detail & Related papers  (2025-06-27T13:59:50Z)
• HandReader: Advanced Techniques for Efficient Fingerspelling Recognition [75.38606213726906]
  This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task.<n>HandReader$_RGB$ employs the novel Adaptive Shift-Temporal Module (TSAM) to process RGB features from videos of varying lengths.<n>HandReader$_KP$ is built on the proposed Temporal Pose (TPE) operated on keypoints as tensors.<n>Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets.
  arXiv  Detail & Related papers  (2025-05-15T13:18:37Z)
• Cross-Modal Learning for Music-to-Music-Video Description Generation [22.27153318775917]
  Music-to-music-video (MV) generation is a challenging task due to intrinsic differences between the music and video modalities. In this study, we focus on the MV description generation task and propose a comprehensive pipeline. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset.
  arXiv  Detail & Related papers  (2025-03-14T08:34:28Z)
• NOTA: Multimodal Music Notation Understanding for Visual Large Language Model [38.26693446133213]
  We propose NOTA, the first large-scale comprehensive multimodal music notation dataset.<n>It consists of 1,019,237 records, from 3 regions of the world, and contains 3 tasks.<n>Based on the dataset, we trained NotaGPT, a music notation visual large language model.
  arXiv  Detail & Related papers  (2025-02-17T16:39:19Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• End-to-end Piano Performance-MIDI to Score Conversion with Transformers [26.900974153235456]
  We present an end-to-end deep learning approach that constructs detailed musical scores directly from real-world piano performance-MIDI files. We introduce a modern transformer-based architecture with a novel tokenized representation for symbolic music data. Our method is also the first to directly predict notational details like trill marks or stem direction from performance data.
  arXiv  Detail & Related papers  (2024-09-30T20:11:37Z)
• AMT-APC: Automatic Piano Cover by Fine-Tuning an Automatic Music Transcription Model [0.0]
  We propose a learning algorithm called AMT-APC, which leverages the capabilities of automatic music transcription models. Our experiments demonstrate that the AMT-APC model reproduces original tracks more accurately than any existing models.
  arXiv  Detail & Related papers  (2024-09-21T09:51:22Z)
• Cluster and Separate: a GNN Approach to Voice and Staff Prediction for Score Engraving [5.572472212662453]
  This paper approaches the problem of separating the notes from a quantized symbolic music piece (e.g., a MIDI file) into multiple voices and staves. We propose an end-to-end system based on graph neural networks that notes that belong to the same chord and connect them with edges if they are part of a voice.
  arXiv  Detail & Related papers  (2024-07-15T14:36:13Z)
• Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation [72.90144343056227]
  We explore the visual representations produced from a pre-trained text-to-video (T2V) diffusion model for video understanding tasks. We introduce a novel framework, termed "VD-IT", tailored with dedicatedly designed components built upon a fixed T2V model. Our VD-IT achieves highly competitive results, surpassing many existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-18T17:59:58Z)
• Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription [13.960714900433269]
  Sheet Music Transformer is the first end-to-end OMR model designed to transcribe complex musical scores without relying solely on monophonic strategies. Our model has been tested on two polyphonic music datasets and has proven capable of handling these intricate music structures effectively.
  arXiv  Detail & Related papers  (2024-02-12T11:52:21Z)
• Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model [83.85856356798531]
  VistaLLM is a visual system that addresses coarse- and fine-grained vision-language tasks. It employs a gradient-aware adaptive sampling technique to represent binary segmentation masks as sequences. We also introduce a novel task, AttCoSeg, which boosts the model's reasoning and grounding capability over multiple input images.
  arXiv  Detail & Related papers  (2023-12-19T18:53:01Z)
• PriViT: Vision Transformers for Fast Private Inference [55.36478271911595]
  Vision Transformer (ViT) architecture has emerged as the backbone of choice for state-of-the-art deep models for computer vision applications. ViTs are ill-suited for private inference using secure multi-party protocols, due to the large number of non-polynomial operations. We propose PriViT, an algorithm to selectively " Taylorize" nonlinearities in ViTs while maintaining their prediction accuracy.
  arXiv  Detail & Related papers  (2023-10-06T21:45:05Z)
• Multitrack Music Transcription with a Time-Frequency Perceiver [6.617487928813374]
  Multitrack music transcription aims to transcribe a music audio input into the musical notes of multiple instruments simultaneously. We propose a novel deep neural network architecture, Perceiver TF, to model the time-frequency representation of audio input for multitrack transcription.
  arXiv  Detail & Related papers  (2023-06-19T08:58:26Z)
• Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction [84.94140661523956]
  We propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. Experiments show that our model trained with sparse supervision effectively predicts the semantic occupancy for all voxels.
  arXiv  Detail & Related papers  (2023-02-15T17:58:10Z)
• RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving [80.14669385741202]
  Vision transformers (ViTs) have achieved state-of-the-art results in many image-based benchmarks. ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. We show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and Semantic KITTI.
  arXiv  Detail & Related papers  (2023-01-24T18:50:48Z)
• Zorro: the masked multimodal transformer [68.99684436029884]
  Zorro is a technique that uses masks to control how inputs from each modality are routed inside Transformers. We show that with contrastive pre-training Zorro achieves state-of-the-art results on most relevant benchmarks for multimodal tasks.
  arXiv  Detail & Related papers  (2023-01-23T17:51:39Z)
• Vision Transformers are Parameter-Efficient Audio-Visual Learners [95.59258503297195]
  We propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks. Our approach achieves competitive or even better performance on various audio-visual tasks.
  arXiv  Detail & Related papers  (2022-12-15T17:31:54Z)
• TVLT: Textless Vision-Language Transformer [89.31422264408002]
  We present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs. TVLT attains performance comparable to its text-based counterpart, on various multimodal tasks. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals.
  arXiv  Detail & Related papers  (2022-09-28T15:08:03Z)
• Polyphonic pitch detection with convolutional recurrent neural networks [0.0]
  In this work, we outline an online polyphonic pitch detection system that streams audio to MIDI by ConvLSTMs. Our system achieves state-of-the-art results on the 2007 MIREX multi-F0 development set, with an F-measure of 83% on the bassoon, clarinet, flute, horn and oboe ensemble recording.
  arXiv  Detail & Related papers  (2022-02-04T12:58:02Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• An Empirical Study of Training End-to-End Vision-and-Language Transformers [50.23532518166621]
  We present METER(textbfMultimodal textbfEnd-to-end textbfTransformtextbfER), through which we investigate how to design and pre-train a fully transformer-based VL model. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion (e.g., merged attention vs. co-
  arXiv  Detail & Related papers  (2021-11-03T17:55:36Z)
• ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction [7.632076006056503]
  ViNet is a fully convolutional encoder-decoder architecture. ViNet does not use audio as input and still outperforms state-of-the-art audio-visual saliency prediction models.
  arXiv  Detail & Related papers  (2020-12-11T07:28:02Z)
• Lets Play Music: Audio-driven Performance Video Generation [58.77609661515749]
  We propose a new task named Audio-driven Per-formance Video Generation (APVG) APVG aims to synthesize the video of a person playing a certain instrument guided by a given music audio clip.
  arXiv  Detail & Related papers  (2020-11-05T03:13:46Z)
• Optical Music Recognition: State of the Art and Major Challenges [0.0]
  Optical Music Recognition (OMR) is concerned with transcribing sheet music into a machine-readable format. The transcribed copy should allow musicians to compose, play and edit music by taking a picture of a music sheet. Recently, there has been a shift in OMR from using conventional computer vision techniques towards a deep learning approach.
  arXiv  Detail & Related papers  (2020-06-14T12:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.