Drum Synthesis from Expressive Drum Grids via Neural Audio Codecs

• URL: http://arxiv.org/abs/2605.10281v1
• Date: Mon, 11 May 2026 09:40:14 GMT
• Title: Drum Synthesis from Expressive Drum Grids via Neural Audio Codecs
• Abstract summary: We propose a system that transforms an expressive drum grid into drum audio by predicting discrete codes of a neural audio.<n>Our approach uses a Transformer-based model to map the drum grid input to a sequence of tokens, which are then converted to waveform audio.<n>We evaluate the fidelity and musical alignment of the generated audio using objective metrics.
• Score: 0.05999777817331315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating realistic drum audio directly from symbolic representations is a challenging task at the intersection of music perception and machine learning. We propose a system that transforms an expressive drum grid, a time-aligned MIDI representation with microtiming and velocity information, into drum audio by predicting discrete codes of a neural audio codec. Our approach uses a Transformer-based model to map the drum grid input to a sequence of codec tokens, which are then converted to waveform audio via a pre-trained codec decoder. We experiment with multiple state-of-the-art neural codecs, namely EnCodec, DAC, and X-Codec, to assess how the choice of audio representation impacts the quality of the generated drums. The system is trained and evaluated on the Expanded Groove MIDI Dataset, E-GMD, a large collection of human drum performances with paired MIDI and audio. We evaluate the fidelity and musical alignment of the generated audio using objective metrics. Overall, our results establish codec-token prediction as an effective route for drum grid-to-audio generation and provide practical insights into selecting audio tokenizers for percussive synthesis.

Related papers

• Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis [18.032312795378004]
  Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts.<n>We introduce Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio.
  arXiv  Detail & Related papers  (2026-05-14T08:32:38Z)
• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• High-Fidelity Music Vocoder using Neural Audio Codecs [18.95453617434051]
  DisCoder is a neural vocoder that reconstructs high-fidelity 44.1 kHz audio from mel spectrograms.<n>DisCoder achieves state-of-the-art performance in music synthesis on several objective metrics and in a MUSHRA listening study.<n>Our approach also shows competitive performance in speech synthesis, highlighting its potential as a universal vocoder.
  arXiv  Detail & Related papers  (2025-02-18T11:25:46Z)
• TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument [19.395289629201056]
  Token Synth is a novel neural synthesizer that generates audio tokens from MIDI tokens and CLAP embedding.<n>Our model is capable of performing instrument cloning, text-to-instrument synthesis, and text-guided timbre manipulation.
  arXiv  Detail & Related papers  (2025-02-13T03:40:30Z)
• Annotation-Free MIDI-to-Audio Synthesis via Concatenative Synthesis and Generative Refinement [0.0]
  CoSaRef is a MIDI-to-audio synthesis method that does not require MIDI-audio paired datasets.<n>It generates a synthetic audio track using concatenative synthesis based on MIDI input, then refines it with a diffusion-based deep generative model trained on datasets without MIDI annotations.<n>It allows detailed control over timbres and expression through audio sample selection and extra MIDI design, similar to traditional functions in digital audio workstations.
  arXiv  Detail & Related papers  (2024-10-22T08:01:40Z)
• Large-scale unsupervised audio pre-training for video-to-speech synthesis [64.86087257004883]
  Video-to-speech synthesis is the task of reconstructing the speech signal from a silent video of a speaker. In this paper we propose to train encoder-decoder models on more than 3,500 hours of audio data at 24kHz. We then use the pre-trained decoders to initialize the audio decoders for the video-to-speech synthesis task.
  arXiv  Detail & Related papers  (2023-06-27T13:31:33Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VA [31.107718069816116]
  We train a Transformer model to improvise the drum part of an unseen drumless recording. We compute beat-related features of the input audio and use them as embeddings in the Transformer. We report both objective and subjective evaluations of variants of the proposed model.
  arXiv  Detail & Related papers  (2022-10-12T08:23:20Z)
• Diffsound: Discrete Diffusion Model for Text-to-sound Generation [78.4128796899781]
  We propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a decoder, and a vocoder. The framework first uses the decoder to transfer the text features extracted from the text encoder to a mel-spectrogram with the help of VQ-VAE, and then the vocoder is used to transform the generated mel-spectrogram into a waveform.
  arXiv  Detail & Related papers  (2022-07-20T15:41:47Z)
• Masked Autoencoders that Listen [79.99280830830854]
  This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram.
  arXiv  Detail & Related papers  (2022-07-13T17:59:55Z)
• DrumGAN VST: A Plugin for Drum Sound Analysis/Synthesis With Autoencoding Generative Adversarial Networks [0.0]
  We present DrumGAN VST, a plugin for synthesizing drum sounds using a Generative Adrial Network. DrumGAN VST operates on 44.1 kHz sample-rate audio, offers independent and continuous instrument class controls, and features an encoding neural network that maps sounds into the GAN's latent space.
  arXiv  Detail & Related papers  (2022-06-29T15:44:19Z)
• Foley Music: Learning to Generate Music from Videos [115.41099127291216]
  Foley Music is a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements.
  arXiv  Detail & Related papers  (2020-07-21T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.