SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

• URL: http://arxiv.org/abs/2604.25498v1
• Date: Tue, 28 Apr 2026 11:01:21 GMT
• Title: SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton
• Abstract summary: SymphonyGen is a 3D hierarchical framework for cinematic orchestration.<n>We introduce "short-score" conditioning via a beat-quantized multi-voice harmony skeleton.<n>We implement a dissonance-averse sampling algorithm to suppress unintended tonal clashes.
• Score: 26.272034375469943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration. Existing symbolic models often struggle with a "complexity-control imbalance", in which scaling bottlenecks limit long-term granular steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary cinematic orchestration. SymphonyGen employs a cascading decoder architecture that decomposes the Bar, Track, and Event axes, improving computational efficiency and scalability over conventional 1D or 2D models. We introduce "short-score" conditioning via a beat-quantized multi-voice harmony skeleton, enabling outline control while preserving textural diversity. The model is further refined using Group Relative Policy Optimization (GRPO) with a cross-modal audio-perceptual reward, aligning symbolic output with modern acoustic expectations. Additionally, we implement a dissonance-averse sampling algorithm to suppress unintended tonal clashes during inference. Objective evaluations show that both reinforcement learning and dissonance-averse sampling effectively enhance harmonic cleanliness while maintaining melodic expression. Subjective evaluations demonstrate that SymphonyGen outperforms baselines in musicality and preference for orchestral music generation. Demo page: https://symphonygen.github.io/

Related papers

• MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music [1.218340575383456]
  We present MIDI-RAE-JEPA, combining a pitch- and time-shift equivariance objective with LeJEPA and a Swin Transformer V2 encoder.<n>We learn such hierarchical representations of symbolic music encoded as piano roll images.<n>Results suggest that equivariance-based SSL objectives, combined with sufficient fine-level encoder capacity, provide a viable path toward semantically rich, generatively useful representations of symbolic music.
  arXiv  Detail & Related papers  (2026-07-16T03:42:35Z)
• Khala: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation [31.997336001394284]
  A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces.<n>In this work, we explore an alternative view: both may be progressively modeled within a single deep acoustic-token hierarchy.<n>Our results suggest that high-quality music generation can be effectively pursued without separating structure and fidelity into heterogeneous representation spaces.
  arXiv  Detail & Related papers  (2026-05-03T09:13:20Z)
• Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation [5.110814852946552]
  We propose the Anchored Cyclic Generation (ACG) paradigm, which relies on anchor features from already identified music to guide subsequent generation.<n>Based on the ACG paradigm, we propose the Hierarchical Anchored Cyclic Generation (Hi-ACG) framework, which is highly compatible with our specifically designed piano token.<n>In long-sequence symbolic music generation tasks, the Hi-ACG framework significantly outperforms existing mainstream methods in both subjective and objective evaluations.
  arXiv  Detail & Related papers  (2026-04-07T02:29:43Z)
• PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation [57.864929968616586]
  Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions.<n>We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning.
  arXiv  Detail & Related papers  (2025-11-24T07:11:12Z)
• Scaling Self-Supervised Representation Learning for Symbolic Piano Performance [52.661197827466886]
  We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions.<n>We use a comparatively smaller, high-quality subset to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings.
  arXiv  Detail & Related papers  (2025-06-30T14:00:14Z)
• SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture [3.7937714754535503]
  SmoothSinger is a conditional diffusion model designed to synthesize high quality and natural singing voices.<n>It refines low-quality synthesized audio directly in a unified framework, mitigating the degradation associated with two-stage pipelines.<n> Experiments on the Opencpop dataset, a large-scale Chinese singing corpus, demonstrate that SmoothSinger achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-26T17:07:45Z)
• MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation [10.203209816178552]
  MotionRAG-Diff is a hybrid framework that integrates Retrieval-Augmented Generation and diffusion-based refinement.<n>Our method introduces three core innovations.<n>It achieves state-of-the-art performance in motion quality, diversity, and music-motion synchronization accuracy.
  arXiv  Detail & Related papers  (2025-06-03T09:12:48Z)
• MuseBarControl: Enhancing Fine-Grained Control in Symbolic Music Generation through Pre-Training and Counterfactual Loss [51.85076222868963]
  We introduce a pre-training task designed to link control signals directly with corresponding musical tokens. We then implement a novel counterfactual loss that promotes better alignment between the generated music and the control prompts.
  arXiv  Detail & Related papers  (2024-07-05T08:08:22Z)
• Symphonize 3D Semantic Scene Completion with Contextual Instance Queries [49.604907627254434]
  3D Semantic Scene Completion (SSC) has emerged as a nascent and pivotal undertaking in autonomous driving. We present a novel paradigm termed Symphonies (Scene-from-Insts), that delves into the integration of instance queries to orchestrate 2D-to-3D reconstruction and 3D scene modeling.
  arXiv  Detail & Related papers  (2023-06-27T17:59:46Z)
• BigVGAN: A Universal Neural Vocoder with Large-Scale Training [49.16254684584935]
  We present BigVGAN, a universal vocoder that generalizes well under various unseen conditions in zero-shot setting. We introduce periodic nonlinearities and anti-aliased representation into the generator, which brings the desired inductive bias for waveform. We train our GAN vocoder at the largest scale up to 112M parameters, which is unprecedented in the literature.
  arXiv  Detail & Related papers  (2022-06-09T17:56:10Z)
• Symphony Generation with Permutation Invariant Language Model [57.75739773758614]
  We present a symbolic symphony music generation solution, SymphonyNet, based on a permutation invariant language model. A novel transformer decoder architecture is introduced as backbone for modeling extra-long sequences of symphony tokens. Our empirical results show that our proposed approach can generate coherent, novel, complex and harmonious symphony compared to human composition.
  arXiv  Detail & Related papers  (2022-05-10T13:08:49Z)
• Deep Performer: Score-to-Audio Music Performance Synthesis [30.95307878579825]
  Deep Performer is a novel system for score-to-audio music performance synthesis. Unlike speech, music often contains polyphony and long notes. We show that our proposed model can synthesize music with clear polyphony and harmonic structures.
  arXiv  Detail & Related papers  (2022-02-12T10:36:52Z)
• Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure [0.0]
  We propose a conditional melody generation model based on a hierarchical recurrent neural network. This model generates melodies with long-term structures based on given chord accompaniments. Results from our listening test indicate that CM-HRNN outperforms AttentionRNN in terms of long-term structure and overall rating.
  arXiv  Detail & Related papers  (2021-02-19T08:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.