COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations

• URL: http://arxiv.org/abs/2404.16969v3
• Date: Wed, 11 Sep 2024 13:23:18 GMT
• Title: COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations
• Authors: Ruben Ciranni, Giorgio Mariani, Michele Mancusi, Emilian Postolache, Giorgio Fabbro, Emanuele Rodolà, Luca Cosmo,
• Abstract summary: COCOLA is a contrastive learning method for musical audio representations that captures the harmonic and rhythmic coherence between samples. Our method operates at the level of the stems composing music tracks and can input features obtained via Harmonic-Percussive Separation (HPS)
• Score: 17.218899140175697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present COCOLA (Coherence-Oriented Contrastive Learning for Audio), a contrastive learning method for musical audio representations that captures the harmonic and rhythmic coherence between samples. Our method operates at the level of the stems composing music tracks and can input features obtained via Harmonic-Percussive Separation (HPS). COCOLA allows the objective evaluation of generative models for music accompaniment generation, which are difficult to benchmark with established metrics. In this regard, we evaluate recent music accompaniment generation models, demonstrating the effectiveness of the proposed method. We release the model checkpoints trained on public datasets containing separate stems (MUSDB18-HQ, MoisesDB, Slakh2100, and CocoChorales).

Related papers

• Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation [3.8570045844185237]
  We present Stem-JEPA, a novel Joint-Embedding Predictive Architecture (JEPA) trained on a multi-track dataset. Our model comprises two networks: an encoder and a predictor, which are jointly trained to predict the embeddings of compatible stems. We evaluate our model's performance on a retrieval task on the MUSDB18 dataset, testing its ability to find the missing stem from a mix.
  arXiv  Detail & Related papers  (2024-08-05T14:34:40Z)
• Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models [2.3749120526936465]
  We propose and investigate the use of neural audio language models for the automatic generation of sample-based musical instruments. Our approach extends a generative audio framework to condition on pitch across an 88-key spectrum, velocity, and a combined text/audio embedding.
  arXiv  Detail & Related papers  (2024-07-22T13:59:58Z)
• CoCoFormer: A controllable feature-rich polyphonic music generation method [2.501600004190393]
  This paper proposes Condition Choir Transformer (CoCoFormer) which controls the output of the model by controlling the chord and rhythm inputs at a fine-grained level. In this paper, the experiments proves that CoCoFormer has reached the current better level than current models.
  arXiv  Detail & Related papers  (2023-10-15T14:04:48Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training [74.32603591331718]
  We propose an acoustic Music undERstanding model with large-scale self-supervised Training (MERT), which incorporates teacher models to provide pseudo labels in the masked language modelling (MLM) style acoustic pre-training. Experimental results indicate that our model can generalise and perform well on 14 music understanding tasks and attain state-of-the-art (SOTA) overall scores.
  arXiv  Detail & Related papers  (2023-05-31T18:27:43Z)
• SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance [88.0355290619761]
  This work focuses on the separation of unknown musical instruments. We propose the Separation-with-Consistency (SeCo) framework, which can accomplish the separation on unknown categories. Our framework exhibits strong adaptation ability on the novel musical categories and outperforms the baseline methods by a significant margin.
  arXiv  Detail & Related papers  (2022-03-25T09:42:11Z)
• Visually Informed Binaural Audio Generation without Binaural Audios [130.80178993441413]
  We propose PseudoBinaural, an effective pipeline that is free of recordings. We leverage spherical harmonic decomposition and head-related impulse response (HRIR) to identify the relationship between spatial locations and received audios. Our-recording-free pipeline shows great stability in cross-dataset evaluation and achieves comparable performance under subjective preference.
  arXiv  Detail & Related papers  (2021-04-13T13:07:33Z)
• Contrastive Learning of General-Purpose Audio Representations [33.15189569532155]
  We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. We build on recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement model of audio.
  arXiv  Detail & Related papers  (2020-10-21T11:56:22Z)
• Unsupervised Cross-Domain Singing Voice Conversion [105.1021715879586]
  We present a wav-to-wav generative model for the task of singing voice conversion from any identity. Our method utilizes both an acoustic model, trained for the task of automatic speech recognition, together with melody extracted features to drive a waveform-based generator.
  arXiv  Detail & Related papers  (2020-08-06T18:29:11Z)
• Score-informed Networks for Music Performance Assessment [64.12728872707446]
  Deep neural network-based methods incorporating score information into MPA models have not yet been investigated. We introduce three different models capable of score-informed performance assessment.
  arXiv  Detail & Related papers  (2020-08-01T07:46:24Z)
• Automatic Melody Harmonization with Triad Chords: A Comparative Study [24.95868747256647]
  We present a comparative study evaluating and comparing the performance of a set of canonical approaches to this task. The evaluation is conducted on a dataset of 9,226 melody/chord pairs we newly collect for this study, considering up to 48 triad chords.
  arXiv  Detail & Related papers  (2020-01-08T03:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.