PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music

• URL: http://arxiv.org/abs/2509.04215v1
• Date: Thu, 04 Sep 2025 13:43:53 GMT
• Title: PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music
• Abstract summary: Solo piano music conveys rich semantic information across genres, moods, and styles.<n>General-purpose music representation models often struggle to capture subtle semantic distinctions within homogeneous solo piano music.<n>We propose PianoBind, a piano-specific multimodal joint embedding model.
• Score: 18.411484855945627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Solo piano music, despite being a single-instrument medium, possesses significant expressive capabilities, conveying rich semantic information across genres, moods, and styles. However, current general-purpose music representation models, predominantly trained on large-scale datasets, often struggle to captures subtle semantic distinctions within homogeneous solo piano music. Furthermore, existing piano-specific representation models are typically unimodal, failing to capture the inherently multimodal nature of piano music, expressed through audio, symbolic, and textual modalities. To address these limitations, we propose PianoBind, a piano-specific multimodal joint embedding model. We systematically investigate strategies for multi-source training and modality utilization within a joint embedding framework optimized for capturing fine-grained semantic distinctions in (1) small-scale and (2) homogeneous piano datasets. Our experimental results demonstrate that PianoBind learns multimodal representations that effectively capture subtle nuances of piano music, achieving superior text-to-music retrieval performance on in-domain and out-of-domain piano datasets compared to general-purpose music joint embedding models. Moreover, our design choices offer reusable insights for multimodal representation learning with homogeneous datasets beyond piano music.

Related papers

• Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping [21.124610627945522]
  We introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation.<n>We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement.<n>Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval.
  arXiv  Detail & Related papers  (2026-08-04T03:01:54Z)
• PianoCoRe: Combined and Refined Piano MIDI Dataset [0.0]
  PianoCoRe is a large-scale piano MIDI dataset that unifies and refines major open-source piano corpora.<n>The dataset contains 250,046 performances of 5,625 pieces written by 483 composers, totaling 21,763 h of performed music.
  arXiv  Detail & Related papers  (2026-05-07T17:41:07Z)
• Music Flamingo: Scaling Music Understanding in Audio Language Models [98.94537017112704]
  Music Flamingo is a novel large audio-language model designed to advance music understanding in foundational audio models.<n> MF-Skills is a dataset labeled through a multi-stage pipeline that yields rich captions and question-answer pairs covering harmony, structure, timbre, lyrics, and cultural context.<n>We introduce a post-training recipe: we first cold-start with MF-Think, a novel chain-of-thought dataset grounded in music theory, followed by GRPO-based reinforcement learning with custom rewards.
  arXiv  Detail & Related papers  (2025-11-13T13:21:09Z)
• Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation [56.318475235705954]
  We present an integrated web toolkit comprising two graphical user interfaces (GUIs)<n>PiaRec supports the synchronized acquisition of audio, video, MIDI, and performance metadata.<n> ASDF enables the efficient annotation of performer fingering from the visual data.
  arXiv  Detail & Related papers  (2025-09-18T17:59:24Z)
• PianoVAM: A Multimodal Piano Performance Dataset [56.318475235705954]
  PianoVAM is a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata.<n>The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions.<n>Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm.
  arXiv  Detail & Related papers  (2025-09-10T17:35:58Z)
• Advancing the Foundation Model for Music Understanding [9.210248657997687]
  We introduce a unified foundation model named MuFun for holistic music understanding.<n>Our model features a novel architecture that jointly processes instrumental and lyrical content.<n>We also propose a new benchmark for multi-faceted music understanding called MuCUE.
  arXiv  Detail & Related papers  (2025-08-02T03:33:47Z)
• Scaling Self-Supervised Representation Learning for Symbolic Piano Performance [52.661197827466886]
  We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions.<n>We use a comparatively smaller, high-quality subset to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings.
  arXiv  Detail & Related papers  (2025-06-30T14:00:14Z)
• From Generality to Mastery: Composer-Style Symbolic Music Generation via Large-Scale Pre-training [4.7205815347741185]
  We investigate how general music knowledge learned from a broad corpus can enhance the mastery of specific composer styles.<n>First, we pre-train a REMI-based music generation model on a large corpus of pop, folk, and classical music.<n>Then, we fine-tune it on a small, human-verified dataset from four renowned composers, namely Bach, Mozart, Beethoven, and Chopin.
  arXiv  Detail & Related papers  (2025-06-20T22:20:59Z)
• Unifying Multitrack Music Arrangement via Reconstruction Fine-Tuning and Efficient Tokenization [10.714947060480426]
  This paper introduces an efficient multitrack music tokenizer for unconditional and conditional symbolic music generation.<n>A unified sequence-to-sequence reconstruction fine-tuning objective for pre-trained music LMs balances task-specific needs with coherence constraints.<n>Our approach achieves state-of-the-art results on band arrangement, piano reduction, and drum arrangement, surpassing task-specific models in both objective metrics and perceptual quality.
  arXiv  Detail & Related papers  (2024-08-27T16:18:51Z)
• PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Training [8.484581633133542]
  PianoBART is a pre-trained model that uses BART for both symbolic piano music generation and understanding. We devise a multi-level object selection strategy for different pre-training tasks of PianoBART, which can prevent information leakage or loss. Experiments demonstrate that PianoBART efficiently learns musical patterns and achieves outstanding performance in generating high-quality coherent pieces.
  arXiv  Detail & Related papers  (2024-06-26T03:35:54Z)
• MARBLE: Music Audio Representation Benchmark for Universal Evaluation [79.25065218663458]
  We introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines.
  arXiv  Detail & Related papers  (2023-06-18T12:56:46Z)
• Generative Modelling for Controllable Audio Synthesis of Expressive Piano Performance [6.531546527140474]
  controllable neural audio synthesizer based on Gaussian Mixture Variational Autoencoders (GM-VAE) We demonstrate how the model is able to apply fine-grained style morphing over the course of the audio.
  arXiv  Detail & Related papers  (2020-06-16T12:54:41Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.