MuSViT: A Foundation Vision Model for Sheet Music Representation

• URL: http://arxiv.org/abs/2606.31811v1
• Date: Tue, 30 Jun 2026 15:27:12 GMT
• Title: MuSViT: A Foundation Vision Model for Sheet Music Representation
• Abstract summary: We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation.<n>MuSViT is a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP.<n>We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification.
• Score: 9.770871275902708
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.

Related papers

• MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music [1.218340575383456]
  We present MIDI-RAE-JEPA, combining a pitch- and time-shift equivariance objective with LeJEPA and a Swin Transformer V2 encoder.<n>We learn such hierarchical representations of symbolic music encoded as piano roll images.<n>Results suggest that equivariance-based SSL objectives, combined with sufficient fine-level encoder capacity, provide a viable path toward semantically rich, generatively useful representations of symbolic music.
  arXiv  Detail & Related papers  (2026-07-16T03:42:35Z)
• LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding [45.71641492661869]
  Legato 2 is a pipeline for extracting symbolic notation and semantic knowledge from images of sheet music.<n>It features the first large-scale neural model for optical music recognition (OMR) to operate sequentially on a system-by-system basis.<n>It is also the first OMR model capable of generating symbolic transcriptions that include embedded textual content.
  arXiv  Detail & Related papers  (2026-07-07T02:47:13Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• Art2Music: Generating Music for Art Images with Multi-modal Feeling Alignment [2.6505592523846495]
  ArtiCaps is a pseudo feeling-aligned image-music-text dataset created by semantically matching descriptions from ArtEmis and MusicCaps.<n>Art2Music is a lightweight cross-modal framework that synthesizes music from artistic images and user comments.<n>Experiments on ArtiCaps show clear improvements in Mel-Cepstral Distortion, Frechet Audio Distance, Log-Spectral Distance, and cosine similarity.
  arXiv  Detail & Related papers  (2025-11-27T21:05:53Z)
• Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music [50.87225308217594]
  This paper presents an unsupervised machine learning algorithm that identifies recurring patterns -- referred to as music-words'' -- from symbolic music data.<n>We formulate the task of music-word discovery as a statistical optimization problem and propose a two-stage Expectation-Maximization (EM)-based learning framework.
  arXiv  Detail & Related papers  (2025-09-29T11:10:57Z)
• Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning [69.78158549955384]
  We introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions.<n>This approach generates verifiable sheet music questions in both textual and visual modalities.<n> Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music.
  arXiv  Detail & Related papers  (2025-09-04T09:42:17Z)
• Exploring Classical Piano Performance Generation with Expressive Music Variational AutoEncoder [15.668253435545921]
  This paper addresses the challenge of generating classical piano performances from scratch, aiming to emulate the dual roles of composer and pianist.<n>We introduce the Expressive Compound Word representation, which effectively captures both the metrical structure and expressive nuances of classical performances.<n>We propose the Expressive Music Variational AutoEncoder (XMVAE), a model featuring two branches: a Vector Quantized Variational AutoEncoder (VQ-VAE) branch that generates score-related content, and a vanilla VAE branch that produces expressive details, fulfilling the role of Pianist.
  arXiv  Detail & Related papers  (2025-07-02T10:54:23Z)
• NOTA: Multimodal Music Notation Understanding for Visual Large Language Model [38.26693446133213]
  We propose NOTA, the first large-scale comprehensive multimodal music notation dataset.<n>It consists of 1,019,237 records, from 3 regions of the world, and contains 3 tasks.<n>Based on the dataset, we trained NotaGPT, a music notation visual large language model.
  arXiv  Detail & Related papers  (2025-02-17T16:39:19Z)
• MuPT: A Generative Symbolic Music Pretrained Transformer [56.09299510129221]
  We explore the application of Large Language Models (LLMs) to the pre-training of music. To address the challenges associated with misaligned measures from different tracks during generation, we propose a Synchronized Multi-Track ABC Notation (SMT-ABC Notation) Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set.
  arXiv  Detail & Related papers  (2024-04-09T15:35:52Z)
• Tr\"aumerAI: Dreaming Music with StyleGAN [2.578242050187029]
  We propose a neural music visualizer directly mapping deep music embeddings to style embeddings of StyleGAN. An annotator listened to 100 music clips of 10 seconds long and selected an image that suits the music among the StyleGAN-generated examples. The generated examples show that the mapping between audio and video makes a certain level of intra-segment similarity and inter-segment dissimilarity.
  arXiv  Detail & Related papers  (2021-02-09T07:04:22Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.