TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure

• URL: http://arxiv.org/abs/2506.23094v1
• Date: Sun, 29 Jun 2025 05:15:41 GMT
• Title: TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure
• Authors: Qi He, Gus Xia, Ziyu Wang,
• Abstract summary: We introduce TOMI (Transforming and Organizing Music Ideas) as a novel approach in deep music generation.<n>We represent a multi-track composition process via a sparse, four-dimensional space characterized by clips (short audio or MIDI segments), sections (temporal positions), tracks (instrument layers) and transformations.<n>Our model is capable of generating multi-track electronic music with full-song structure, and we further integrate the TOMI-based model with the REAPER digital audio workstation.
• Score: 8.721294663967305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hierarchical planning is a powerful approach to model long sequences structurally. Aside from considering hierarchies in the temporal structure of music, this paper explores an even more important aspect: concept hierarchy, which involves generating music ideas, transforming them, and ultimately organizing them--across musical time and space--into a complete composition. To this end, we introduce TOMI (Transforming and Organizing Music Ideas) as a novel approach in deep music generation and develop a TOMI-based model via instruction-tuned foundation LLM. Formally, we represent a multi-track composition process via a sparse, four-dimensional space characterized by clips (short audio or MIDI segments), sections (temporal positions), tracks (instrument layers), and transformations (elaboration methods). Our model is capable of generating multi-track electronic music with full-song structure, and we further integrate the TOMI-based model with the REAPER digital audio workstation, enabling interactive human-AI co-creation. Experimental results demonstrate that our approach produces higher-quality electronic music with stronger structural coherence compared to baselines.

Related papers

• Advancing the Foundation Model for Music Understanding [9.210248657997687]
  We introduce a unified foundation model named MuFun for holistic music understanding.<n>Our model features a novel architecture that jointly processes instrumental and lyrical content.<n>We also propose a new benchmark for multi-faceted music understanding called MuCUE.
  arXiv  Detail & Related papers  (2025-08-02T03:33:47Z)
• Synthesizing Composite Hierarchical Structure from Symbolic Music Corpora [32.18458296933001]
  We propose a unified, hierarchical meta-representation of musical structure called the structural temporal graph (STG)<n>For a single piece, the STG is a data structure that defines a hierarchy of progressively finer structural musical features and the temporal relationships between them.
  arXiv  Detail & Related papers  (2025-02-21T02:32:29Z)
• UniMuMo: Unified Text, Music and Motion Generation [57.72514622935806]
  We introduce UniMuMo, a unified multimodal model capable of taking arbitrary text, music, and motion data as input conditions to generate outputs across all three modalities. By converting music, motion, and text into token-based representation, our model bridges these modalities through a unified encoder-decoder transformer architecture.
  arXiv  Detail & Related papers  (2024-10-06T16:04:05Z)
• Foundation Models for Music: A Survey [77.77088584651268]
  Foundations models (FMs) have profoundly impacted diverse sectors, including music. This comprehensive review examines state-of-the-art (SOTA) pre-trained models and foundation models in music.
  arXiv  Detail & Related papers  (2024-08-26T15:13:14Z)
• ComposerX: Multi-Agent Symbolic Music Composition with LLMs [51.68908082829048]
  Music composition is a complex task that requires abilities to understand and generate information with long dependency and harmony constraints. Current LLMs easily fail in this task, generating ill-written music even when equipped with modern techniques like In-Context-Learning and Chain-of-Thoughts. We propose ComposerX, an agent-based symbolic music generation framework.
  arXiv  Detail & Related papers  (2024-04-28T06:17:42Z)
• MuPT: A Generative Symbolic Music Pretrained Transformer [56.09299510129221]
  We explore the application of Large Language Models (LLMs) to the pre-training of music. To address the challenges associated with misaligned measures from different tracks during generation, we propose a Synchronized Multi-Track ABC Notation (SMT-ABC Notation) Our contributions include a series of models capable of handling up to 8192 tokens, covering 90% of the symbolic music data in our training set.
  arXiv  Detail & Related papers  (2024-04-09T15:35:52Z)
• ByteComposer: a Human-like Melody Composition Method based on Language Model Agent [11.792129708566598]
  Large Language Models (LLM) have shown encouraging progress in multimodal understanding and generation tasks. We propose ByteComposer, an agent framework emulating a human's creative pipeline in four separate steps. We conduct extensive experiments on GPT4 and several open-source large language models, which substantiate our framework's effectiveness.
  arXiv  Detail & Related papers  (2024-02-24T04:35:07Z)
• Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generation [50.365392018302416]
  We propose Multi-view MidiVAE, as one of the pioneers in VAE methods that effectively model and generate long multi-track symbolic music. We focus on instrumental characteristics and harmony as well as global and local information about the musical composition by employing a hybrid variational encoding-decoding strategy.
  arXiv  Detail & Related papers  (2024-01-15T08:41:01Z)
• Graph-based Polyphonic Multitrack Music Generation [9.701208207491879]
  This paper introduces a novel graph representation for music and a deep Variational Autoencoder that generates the structure and the content of musical graphs separately. By separating the structure and content of musical graphs, it is possible to condition generation by specifying which instruments are played at certain times.
  arXiv  Detail & Related papers  (2023-07-27T15:18:50Z)
• The Power of Reuse: A Multi-Scale Transformer Model for Structural Dynamic Segmentation in Symbolic Music Generation [6.0949335132843965]
  Symbolic Music Generation relies on the contextual representation capabilities of the generative model. We propose a multi-scale Transformer, which uses coarse-decoder and fine-decoders to model the contexts at the global and section-level. Our model is evaluated on two open MIDI datasets, and experiments show that our model outperforms the best contemporary symbolic music generative models.
  arXiv  Detail & Related papers  (2022-05-17T18:48:14Z)
• Lets Play Music: Audio-driven Performance Video Generation [58.77609661515749]
  We propose a new task named Audio-driven Per-formance Video Generation (APVG) APVG aims to synthesize the video of a person playing a certain instrument guided by a given music audio clip.
  arXiv  Detail & Related papers  (2020-11-05T03:13:46Z)
• Modeling Musical Structure with Artificial Neural Networks [0.0]
  I explore the application of artificial neural networks to different aspects of musical structure modeling. I show how a connectionist model, the Gated Autoencoder (GAE), can be employed to learn transformations between musical fragments. I propose a special predictive training of the GAE, which yields a representation of polyphonic music as a sequence of intervals.
  arXiv  Detail & Related papers  (2020-01-06T18:35:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.