Multi-Task Multi-Frame Visual Piano Transcription

• URL: http://arxiv.org/abs/2608.03419v1
• Date: Tue, 04 Aug 2026 10:11:45 GMT
• Title: Multi-Task Multi-Frame Visual Piano Transcription
• Abstract summary: We present V2N (Video to Notes), the first complete Visual Piano Transcription system.<n>A shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity.<n>Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy.
• Score: 57.32059378506694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.

Related papers

• Surprise Forcing: What to Remember, When to Skip in Long Video Generation [65.0022589149937]
  Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly.<n>We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems.<n> Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.
  arXiv  Detail & Related papers  (2026-07-20T18:37:58Z)
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies [58.65913948991329]
  We present VISUALTHINK-VLA, a visual intermediate-reasoning framework for accurate, low-latency VLA policies.<n>Our bootstrapping philosophy is to guide action with effective visual thinking.<n>It bootstraps action prediction through a compact visual-evidence interface that preserves spatial precision while avoiding decoding overhead.
  arXiv  Detail & Related papers  (2026-05-28T14:36:53Z)
• VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models [80.1055544841585]
  Vision-Language-Action models typically map visual observations and linguistic instructions directly to robotic control signals.<n>We propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface.<n> Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%.
  arXiv  Detail & Related papers  (2026-03-23T14:08:58Z)
• Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations [2.3249139042158853]
  We present a neural network approach for synchronizing audio recordings of human piano performances with their corresponding loosely aligned MIDI files.<n>The proposed model achieves up to 20% higher alignment accuracy than the industry-standard Dynamic Time Warping (DTW) method.
  arXiv  Detail & Related papers  (2025-06-27T13:59:50Z)
• CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling [84.51372201195132]
  CronusVLA is a unified framework that extends single-frame VLA models to the multi-frame paradigm.<n>CronusVLA achieves leading performance and superior robustness, with a 70.9% success rate.<n>These results highlight the potential of efficient multi-frame adaptation in VLA models for more powerful and robust real-world deployment.
  arXiv  Detail & Related papers  (2025-06-24T17:30:27Z)
• Exploring Temporally-Aware Features for Point Tracking [58.63091479730935]
  Chrono is a feature backbone specifically designed for point tracking with built-in temporal awareness.<n>Chrono achieves state-of-the-art performance in a refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets.
  arXiv  Detail & Related papers  (2025-01-21T15:39:40Z)
• FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding [25.21011724370177]
  Text-guided Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on textual descriptions.<n>We introduce FlashVTG, a framework featuring a Temporal Feature Layering (TFL) module and an Adaptive Score Refinement (ASR) module.<n>FlashVTG achieves state-of-the-art performance on four widely adopted datasets in both Moment Retrieval (MR) and Highlight Detection (HD)
  arXiv  Detail & Related papers  (2024-12-18T02:23:33Z)
• Pay Attention to the Keys: Visual Piano Transcription Using Transformers [0.0]
  Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone.<n>We propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs)<n>We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets.
  arXiv  Detail & Related papers  (2024-11-13T21:31:12Z)
• Temporal Pyramid Network for Action Recognition [129.12076009042622]
  We propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks. TPN shows consistent improvements over other challenging baselines on several action recognition datasets.
  arXiv  Detail & Related papers  (2020-04-07T17:17:23Z)
• Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network [35.932447204088845]
  We develop an effective feature alignment network tailored to video saliency prediction (V) The network learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner. The proposed model is trained end-to-end without any post processing.
  arXiv  Detail & Related papers  (2020-01-02T02:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.