STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts

• URL: http://arxiv.org/abs/2605.12135v1
• Date: Tue, 12 May 2026 13:56:09 GMT
• Title: STRUM: A Spectral Transcription and Rhythm Understanding Model for End-to-End Generation of Playable Rhythm-Game Charts
• Abstract summary: We present STRUM, an audio-to-chart pipeline that converts raw recordings into playable Clone Hero charts for drums, guitar, bass, vocals, and keys without any oracle metadata.<n>We report a complete ablation of seven drum-pipeline components with paired per-song Wilcoxon tests, an analysis of ground-truth-to-audio timing distributions in community Clone Hero charts, and a per-class confusion matrix for the drum.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present STRUM (Spectral Transcription and Rhythm Understanding Model), an audio-to-chart pipeline that converts raw recordings into playable Clone Hero / YARG charts for drums, guitar, bass, vocals, and keys without any oracle metadata. STRUM is a multi-stage hybrid: a two-stage CRNN onset detector and a six-model ensemble classifier for drums; neural onset detectors with monophonic pitch tracking for guitar and bass; word-aligned ASR for vocals; and spectral keyboard detection for keys. We evaluate on a 30-song in-envelope benchmark constructed by screening candidate songs on a single audio-quality criterion -- the median 1-second drum-stem RMS after htdemucs_6s source separation. On this benchmark STRUM achieves drums onset F1 = 0.838, bass F1 = 0.694, guitar F1 = 0.651, and vocals F1 = 0.539 at a +/- 100 ms tolerance with per-song global offset search. We report a complete ablation of seven drum-pipeline components with paired per-song Wilcoxon tests, an analysis of ground-truth-to-audio timing distributions in community Clone Hero charts, and a per-class confusion matrix for the drum classifier. Code, model weights, and the full benchmark manifest are released.

Related papers

• Fretiq: Browser-Native Electric Guitar String Classification via Engineered Spectral Features and Held-Out Free-Play Evaluation [0.0]
  Fretiq is a preliminary single-instrument, single-player browser-based string classification system.<n>It achieves 97.1% shuffled frame-level validation accuracy across 322,215 balanced frames.<n>System runs entirely in-browser with no hexaphonic pickup, fretboard sensor, camera, or multi-microphone setup required.
  arXiv  Detail & Related papers  (2026-07-17T03:06:07Z)
• How Far Can Chord-Symbol Time-Series Adaptation Carry Genre Identity? Capabilities and Boundaries in Multi-Genre Chord-Symbol Modeling [0.0]
  This report treats chord-symbol sequences as an interpretable, controllable time series for genre-local harmonic modeling.<n>Main evaluation compares LoRA, IA3, BitFit, prefix tuning, and full fine-tuning over 11 genres and 3 seeds, a complete 165-cell grid.<n> chord-symbol adaptation reliably improves genre-local harmonic prediction, but chord symbols alone do not carry complete genre identity.
  arXiv  Detail & Related papers  (2026-06-05T14:49:24Z)
• Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis [18.032312795378004]
  Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts.<n>We introduce Break-the-Beat!,'' a model capable of rendering a drum MIDI with the timbre of a reference audio.
  arXiv  Detail & Related papers  (2026-05-14T08:32:38Z)
• FlashSampling: Fast and Memory-Efficient Exact Sampling [62.5203057469482]
  FlashSampling is an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in.<n>Across H100, H200, B200, and B300 GPUs, FlashSampling speeds up kernel-level decode workloads.<n>In end-to-end vLLM experiments, it reduces time per output token by up to $19%$ on the models we test.
  arXiv  Detail & Related papers  (2026-03-16T19:37:08Z)
• Towards Realistic Synthetic Data for Automatic Drum Transcription [3.975380931806995]
  This paper introduces a new paradigm for Automatic Drum Transcription (ADT) that circumvents the need for paired audio-MIDI training data.<n>Our primary contribution is a semi-supervised method to automatically curate a large and diverse corpus of one-shot drum samples from unlabeled audio sources.<n>We then use this corpus to synthesize a high-quality dataset from MIDI files alone, which we use to train a sequence-to-sequence transcription model.
  arXiv  Detail & Related papers  (2026-01-14T14:39:05Z)
• Barwise Section Boundary Detection in Symbolic Music Using Convolutional Neural Networks [0.0]
  We introduce a human-annotated MIDI dataset for section boundary detection.<n>Second, we train a deep learning model to classify the presence of section boundaries within a fixed-length musical window.<n>Our model achieves an F1 score of 0.77, improving over the analogous audio-based supervised learning approach.
  arXiv  Detail & Related papers  (2025-09-20T07:52:08Z)
• EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning [108.73513190593232]
  Multimodal large language models (MLLMs) have advanced perception across text, vision, and audio, yet struggle with structured cross-modal reasoning.<n>We introduce EchoInk-R1, a reinforcement learning framework that enhances such reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-05-07T17:59:49Z)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation.<n>It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately.<n>To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• Toward Deep Drum Source Separation [52.01259769265708]
  We introduce StemGMD, a large-scale audio dataset of isolated single-instrument drum stems. Totaling 1224 hours, StemGMD is the largest audio dataset of drums to date. We leverage StemGMD to develop LarsNet, a novel deep drum source separation model.
  arXiv  Detail & Related papers  (2023-12-15T10:23:07Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• Benchmarks and leaderboards for sound demixing tasks [44.99833362998488]
  We introduce two new benchmarks for the sound source separation tasks. We compare popular models for sound demixing, as well as their ensembles, on these benchmarks. We also develop a novel approach for audio separation, based on the ensembling of different models that are suited best for the particular stem.
  arXiv  Detail & Related papers  (2023-05-12T14:00:26Z)
• AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment [67.10208647482109]
  The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment. Experiments show that AlignSTS achieves superior performance in terms of both objective and subjective metrics.
  arXiv  Detail & Related papers  (2023-05-08T06:02:10Z)
• GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers [14.025337055088102]
  We use the DadaGP dataset for guitar tab music generation, a corpus of over 26k songs in GuitarPro and token formats. We introduce methods to condition a Transformer-XL deep learning model to generate guitar tabs based on desired instrumentation and genre. Results indicate that the GTR-CTRL methods provide more flexibility and control for guitar-focused symbolic music generation than an unconditioned model.
  arXiv  Detail & Related papers  (2023-02-10T17:43:03Z)
• Multi-instrument Music Synthesis with Spectrogram Diffusion [19.81982315173444]
  We focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.
  arXiv  Detail & Related papers  (2022-06-11T03:26:15Z)
• Conditional Drums Generation using Compound Word Representations [4.435094091999926]
  We tackle the task of conditional drums generation using a novel data encoding scheme inspired by Compound Word representation. We present a sequence-to-sequence architecture where a Bidirectional Long short-term memory (BiLSTM) receives information about the conditioning parameters. A Transformer-based Decoder with relative global attention produces the generated drum sequences.
  arXiv  Detail & Related papers  (2022-02-09T13:49:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.