Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis

• URL: http://arxiv.org/abs/2605.14555v1
• Date: Thu, 14 May 2026 08:32:38 GMT
• Title: Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis
• Abstract summary: Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts.<n>We introduce Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio.
• Score: 18.032312795378004
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts of creators. While recent generative models achieve high fidelity and adhere to text, they lack the specific control needed for such a task. Existing symbolic-to-audio research often focuses on single, tonal instruments, leaving the challenge of polyphonic, percussive drum synthesis unaddressed. We address this gap by introducing ``Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio. It is built by fine-tuning a pre-trained text-to-audio model with our proposed content encoder and a effective hybrid conditioning mechanism. To enable this, we construct a new dataset of paired target-reference drum audio from existing drum audio datasets. Experiments demonstrate that our model generates high-quality drum audio that follows high-resolution drum MIDI, achieving strong performance across metrics of audio quality, rhythmic alignment, and beat continuity. This offer producers a new, controllable tool for creative production. Demo page: https://ik4sumii.github.io/break-the-beat/

Related papers

• Drum Synthesis from Expressive Drum Grids via Neural Audio Codecs [0.05999777817331315]
  We propose a system that transforms an expressive drum grid into drum audio by predicting discrete codes of a neural audio.<n>Our approach uses a Transformer-based model to map the drum grid input to a sequence of tokens, which are then converted to waveform audio.<n>We evaluate the fidelity and musical alignment of the generated audio using objective metrics.
  arXiv  Detail & Related papers  (2026-05-11T09:40:14Z)
• Towards Realistic Synthetic Data for Automatic Drum Transcription [3.975380931806995]
  This paper introduces a new paradigm for Automatic Drum Transcription (ADT) that circumvents the need for paired audio-MIDI training data.<n>Our primary contribution is a semi-supervised method to automatically curate a large and diverse corpus of one-shot drum samples from unlabeled audio sources.<n>We then use this corpus to synthesize a high-quality dataset from MIDI files alone, which we use to train a sequence-to-sequence transcription model.
  arXiv  Detail & Related papers  (2026-01-14T14:39:05Z)
• Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control [66.46754271097555]
  We release a fully open-source system for long-form song generation with fine-grained style conditioning.<n>The dataset consists of 116k fully licensed synthetic songs with automatically generated lyrics and style descriptions.<n>We train Muse via single-stage supervised finetuning of a Qwen-based language model extended with discrete audio tokens.
  arXiv  Detail & Related papers  (2026-01-07T14:40:48Z)
• Music Boomerang: Reusing Diffusion Models for Data Augmentation and Audio Manipulation [49.062766449989525]
  Generative models of music audio are typically used to generate output based solely on a text prompt or melody.<n>Boomerang sampling, recently proposed for the image domain, allows generating output close to an existing example, using any pretrained diffusion model.
  arXiv  Detail & Related papers  (2025-07-07T10:46:07Z)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation.<n>It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately.<n>To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement [0.0]
  CoSaRef is a MIDI-to-audio synthesis method that does not require MIDI-audio paired datasets.<n>It generates a synthetic audio track using concatenative synthesis based on MIDI input, then refines it with a diffusion-based deep generative model trained on datasets without MIDI annotations.<n>It allows detailed control over timbres and expression through audio sample selection and extra MIDI design, similar to traditional functions in digital audio workstations.
  arXiv  Detail & Related papers  (2024-10-22T08:01:40Z)
• Toward Deep Drum Source Separation [52.01259769265708]
  We introduce StemGMD, a large-scale audio dataset of isolated single-instrument drum stems. Totaling 1224 hours, StemGMD is the largest audio dataset of drums to date. We leverage StemGMD to develop LarsNet, a novel deep drum source separation model.
  arXiv  Detail & Related papers  (2023-12-15T10:23:07Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• Benchmarks and leaderboards for sound demixing tasks [44.99833362998488]
  We introduce two new benchmarks for the sound source separation tasks. We compare popular models for sound demixing, as well as their ensembles, on these benchmarks. We also develop a novel approach for audio separation, based on the ensembling of different models that are suited best for the particular stem.
  arXiv  Detail & Related papers  (2023-05-12T14:00:26Z)
• JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VA [31.107718069816116]
  We train a Transformer model to improvise the drum part of an unseen drumless recording. We compute beat-related features of the input audio and use them as embeddings in the Transformer. We report both objective and subjective evaluations of variants of the proposed model.
  arXiv  Detail & Related papers  (2022-10-12T08:23:20Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• DrumGAN VST: A Plugin for Drum Sound Analysis/Synthesis With Autoencoding Generative Adversarial Networks [0.0]
  We present DrumGAN VST, a plugin for synthesizing drum sounds using a Generative Adrial Network. DrumGAN VST operates on 44.1 kHz sample-rate audio, offers independent and continuous instrument class controls, and features an encoding neural network that maps sounds into the GAN's latent space.
  arXiv  Detail & Related papers  (2022-06-29T15:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.