LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

• URL: http://arxiv.org/abs/2607.05769v1
• Date: Tue, 07 Jul 2026 02:47:13 GMT
• Title: LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding
• Abstract summary: Legato 2 is a pipeline for extracting symbolic notation and semantic knowledge from images of sheet music.<n>It features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis.<n>It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content.
• Score: 45.71641492661869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music. Legato 2 features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis, following the horizontal lines of notation as they are read on the page, rather than treating the page as an undifferentiated image, enabling better scaling to arbitrarily long inputs. It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content, such as titles and annotations. The pipeline combines system-level segmentation with an autoregressive vision-LM to capture both local notation details and score structure. Across multiple datasets, Legato 2 consistently outperforms prior state of the art. We also show that symbolic transcriptions complement visual inputs for frontier language models, improving their interpretation of dense musical documents. Legato 2 establishes new state-of-the-art performance in both OMR and downstream sheet music understanding.

Related papers

• MuSViT: A Foundation Vision Model for Sheet Music Representation [9.770871275902708]
  We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation.<n>MuSViT is a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP.<n>We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification.
  arXiv  Detail & Related papers  (2026-06-30T15:27:12Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• From Image to Music Language: A Two-Stage Structure Decoding Approach for Complex Polyphonic OMR [23.454234180354323]
  We propose a new approach for a practical two-stage Optical Music Recognition (OMR) pipeline.<n>Given symbol and event candidates from the visual pipeline, we decode them into an editable, verifiable, and exportable score structure.<n>We focus on complex polyphonic staff notation, especially piano scores, where voice separation and intra-measure timing are the main bottlenecks.
  arXiv  Detail & Related papers  (2026-04-22T13:01:36Z)
• FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing [81.3306413498174]
  Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects.<n>Existing methods focus primarily on reducing the word error rate while ignoring the importance of lip-sync and acoustic quality.<n>We propose a large language model (LLM) based flow matching architecture for dubbing, named FlowDubber.
  arXiv  Detail & Related papers  (2025-05-02T13:30:19Z)
• NOTA: Multimodal Music Notation Understanding for Visual Large Language Model [38.26693446133213]
  We propose NOTA, the first large-scale comprehensive multimodal music notation dataset.<n>It consists of 1,019,237 records, from 3 regions of the world, and contains 3 tasks.<n>Based on the dataset, we trained NotaGPT, a music notation visual large language model.
  arXiv  Detail & Related papers  (2025-02-17T16:39:19Z)
• Toward a More Complete OMR Solution [49.74172035862698]
  Optical music recognition aims to convert music notation into digital formats. One approach to tackle OMR is through a multi-stage pipeline, where the system first detects visual music notation elements in the image. We introduce a music object detector based on YOLOv8, which improves detection performance. Second, we introduce a supervised training pipeline that completes the notation assembly stage based on detection output.
  arXiv  Detail & Related papers  (2024-08-31T01:09:12Z)
• Knowledge Discovery in Optical Music Recognition: Enhancing Information Retrieval with Instance Segmentation [0.0]
  Optical Music Recognition (OMR) automates the transcription of musical notation from images into machine-readable formats like MusicXML, MEI, or MIDI. This study explores knowledge discovery in OMR by applying instance segmentation using Mask R-CNN to enhance the detection and delineation of musical symbols in sheet music.
  arXiv  Detail & Related papers  (2024-08-27T12:34:41Z)
• End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music [12.779526750915707]
  We present the first truly end-to-end approach for page-level Optical Music Recognition in complex layouts.<n>Our system, which combines convolutional layers with autoregressive Transformers, processes an entire music score page and outputs a complete transcription in a music encoding format.<n>The results demonstrate that our system not only successfully transcribes full-page music scores but also outperforms the commercial tool in both zero-shot settings and after fine-tuning with the target domain.
  arXiv  Detail & Related papers  (2024-05-20T15:21:48Z)
• Single-Stream Multi-Level Alignment for Vision-Language Pretraining [103.09776737512078]
  We propose a single stream model that aligns the modalities at multiple levels. We achieve this using two novel tasks: symmetric cross-modality reconstruction and a pseudo-labeled key word prediction. We demonstrate top performance on a set of Vision-Language downstream tasks such as zero-shot/fine-tuned image/text retrieval, referring expression, and VQA.
  arXiv  Detail & Related papers  (2022-03-27T21:16:10Z)
• End-to-end Audio-visual Speech Recognition with Conformers [65.30276363777514]
  We present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer) In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms. We show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.
  arXiv  Detail & Related papers  (2021-02-12T18:00:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.