CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models

• URL: http://arxiv.org/abs/2410.13267v1
• Date: Thu, 17 Oct 2024 06:43:54 GMT
• Title: CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models
• Authors: Shangda Wu, Yashan Wang, Ruibin Yuan, Zhancheng Guo, Xu Tan, Ge Zhang, Monan Zhou, Jing Chen, Xuefeng Mu, Yuejie Gao, Yuanliang Dong, Jiafeng Liu, Xiaobing Li, Feng Yu, Maosong Sun,
• Abstract summary: CLaMP 2 is a system compatible with 101 languages for music information retrieval. By leveraging large language models, we obtain refined and consistent multilingual descriptions at scale. CLaMP 2 achieves state-of-the-art results in both multilingual semantic search and music classification across modalities.
• Score: 51.03510073676228
• License:
• Abstract: Challenges in managing linguistic diversity and integrating various musical modalities are faced by current music information retrieval systems. These limitations reduce their effectiveness in a global, multimodal music environment. To address these issues, we introduce CLaMP 2, a system compatible with 101 languages that supports both ABC notation (a text-based musical notation format) and MIDI (Musical Instrument Digital Interface) for music information retrieval. CLaMP 2, pre-trained on 1.5 million ABC-MIDI-text triplets, includes a multilingual text encoder and a multimodal music encoder aligned via contrastive learning. By leveraging large language models, we obtain refined and consistent multilingual descriptions at scale, significantly reducing textual noise and balancing language distribution. Our experiments show that CLaMP 2 achieves state-of-the-art results in both multilingual semantic search and music classification across modalities, thus establishing a new standard for inclusive and global music information retrieval.

Related papers

• MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models [11.834712543531756]
  MuChoMusic is a benchmark for evaluating music understanding in multimodal language models focused on audio. It comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets. We evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality.
  arXiv  Detail & Related papers  (2024-08-02T15:34:05Z)
• MuPT: A Generative Symbolic Music Pretrained Transformer [56.09299510129221]
  We explore the application of Large Language Models (LLMs) to the pre-training of music. To address the challenges associated with misaligned measures from different tracks during generation, we propose a Synchronized Multi-Track ABC Notation (SMT-ABC Notation) Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set.
  arXiv  Detail & Related papers  (2024-04-09T15:35:52Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• ChatMusician: Understanding and Generating Music Intrinsically with LLM [81.48629006702409]
  ChatMusician is an open-source Large Language Models (LLMs) that integrates intrinsic musical abilities. It can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc.
  arXiv  Detail & Related papers  (2024-02-25T17:19:41Z)
• Scaling Speech Technology to 1,000+ Languages [66.31120979098483]
  The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. Main ingredients are a new dataset based on readings of publicly available religious texts. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, and a language identification model for 4,017 languages.
  arXiv  Detail & Related papers  (2023-05-22T22:09:41Z)
• LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation [94.33019040320507]
  Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases. We propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.
  arXiv  Detail & Related papers  (2022-10-19T12:21:39Z)
• MuLan: A Joint Embedding of Music Audio and Natural Language [15.753767984842014]
  This paper presents a new generation of models that link audio annotations directly to natural language descriptions. MuLan takes the form of a two-tower, joint audio-text embedding model trained using 44 million music recordings.
  arXiv  Detail & Related papers  (2022-08-26T03:13:21Z)
• Contrastive Audio-Language Learning for Music [13.699088044513562]
  MusCALL is a framework for Music Contrastive Audio-Language Learning. Our approach consists of a dual-encoder architecture that learns the alignment between pairs of music audio and descriptive sentences.
  arXiv  Detail & Related papers  (2022-08-25T16:55:15Z)
• Multilingual Music Genre Embeddings for Effective Cross-Lingual Music Item Annotation [9.709229853995987]
  Cross-lingual music genre translation is possible without relying on a parallel corpus. By learning multilingual music genre embeddings, we enable cross-lingual music genre translation without relying on a parallel corpus. Our method is effective in translating music genres across tag systems in multiple languages.
  arXiv  Detail & Related papers  (2020-09-16T15:39:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.