PerceiverS: A Multi-Scale Perceiver with Effective Segmentation for Long-Term Expressive Symbolic Music Generation

• URL: http://arxiv.org/abs/2411.08307v1
• Date: Wed, 13 Nov 2024 03:14:10 GMT
• Title: PerceiverS: A Multi-Scale Perceiver with Effective Segmentation for Long-Term Expressive Symbolic Music Generation
• Authors: Yungang Yi, Weihua Li, Matthew Kuo, Quan Bai,
• Abstract summary: PerceiverS (Segmentation and Scale) is a novel architecture designed to generate long-structured and expressive music. Our approach enhances symbolic music generation by simultaneously learning long-term structural dependencies and short-term expressive details. The proposed model, evaluated on datasets like Maestro, demonstrates improvements in generating coherent and diverse music.
• Score: 5.201151187019607
• License:
• Abstract: Music generation has progressed significantly, especially in the domain of audio generation. However, generating symbolic music that is both long-structured and expressive remains a significant challenge. In this paper, we propose PerceiverS (Segmentation and Scale), a novel architecture designed to address this issue by leveraging both Effective Segmentation and Multi-Scale attention mechanisms. Our approach enhances symbolic music generation by simultaneously learning long-term structural dependencies and short-term expressive details. By combining cross-attention and self-attention in a Multi-Scale setting, PerceiverS captures long-range musical structure while preserving performance nuances. The proposed model, evaluated on datasets like Maestro, demonstrates improvements in generating coherent and diverse music with both structural consistency and expressive variation. The project demos and the generated music samples can be accessed through the link: https://perceivers.github.io.

Related papers

• MuPT: A Generative Symbolic Music Pretrained Transformer [56.09299510129221]
  We explore the application of Large Language Models (LLMs) to the pre-training of music. To address the challenges associated with misaligned measures from different tracks during generation, we propose a Synchronized Multi-Track ABC Notation (SMT-ABC Notation) Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set.
  arXiv  Detail & Related papers  (2024-04-09T15:35:52Z)
• Structure-informed Positional Encoding for Music Generation [0.0]
  We propose a structure-informed positional encoding framework for music generation with Transformers. We test them on two symbolic music generation tasks: next-timestep prediction and accompaniment generation. Our methods improve the melodic and structural consistency of the generated pieces.
  arXiv  Detail & Related papers  (2024-02-20T13:41:35Z)
• Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generation [50.365392018302416]
  We propose Multi-view MidiVAE, as one of the pioneers in VAE methods that effectively model and generate long multi-track symbolic music. We focus on instrumental characteristics and harmony as well as global and local information about the musical composition by employing a hybrid variational encoding-decoding strategy.
  arXiv  Detail & Related papers  (2024-01-15T08:41:01Z)
• Graph-based Polyphonic Multitrack Music Generation [9.701208207491879]
  This paper introduces a novel graph representation for music and a deep Variational Autoencoder that generates the structure and the content of musical graphs separately. By separating the structure and content of musical graphs, it is possible to condition generation by specifying which instruments are played at certain times.
  arXiv  Detail & Related papers  (2023-07-27T15:18:50Z)
• MARBLE: Music Audio Representation Benchmark for Universal Evaluation [79.25065218663458]
  We introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines.
  arXiv  Detail & Related papers  (2023-06-18T12:56:46Z)
• Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation [138.74751744348274]
  We propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the tokens of the bars that are most relevant to music structures. With the coarse-grained attention, a token only attends to the summarization of the other bars rather than each token of them so as to reduce the computational cost.
  arXiv  Detail & Related papers  (2022-10-19T07:31:56Z)
• MATT: A Multiple-instance Attention Mechanism for Long-tail Music Genre Classification [1.8275108630751844]
  Imbalanced music genre classification is a crucial task in the Music Information Retrieval (MIR) field. Most of the existing models are designed for class-balanced music datasets. We propose a novel mechanism named Multi-instance Attention (MATT) to boost the performance for identifying tail classes.
  arXiv  Detail & Related papers  (2022-09-09T03:52:44Z)
• Lets Play Music: Audio-driven Performance Video Generation [58.77609661515749]
  We propose a new task named Audio-driven Per-formance Video Generation (APVG) APVG aims to synthesize the video of a person playing a certain instrument guided by a given music audio clip.
  arXiv  Detail & Related papers  (2020-11-05T03:13:46Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)
• Modeling Musical Structure with Artificial Neural Networks [0.0]
  I explore the application of artificial neural networks to different aspects of musical structure modeling. I show how a connectionist model, the Gated Autoencoder (GAE), can be employed to learn transformations between musical fragments. I propose a special predictive training of the GAE, which yields a representation of polyphonic music as a sequence of intervals.
  arXiv  Detail & Related papers  (2020-01-06T18:35:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.