Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation

• URL: http://arxiv.org/abs/2606.24307v1
• Date: Tue, 23 Jun 2026 08:37:15 GMT
• Title: Real-Time Interactive Music Generation via Data-Free Streaming Consistency Distillation
• Abstract summary: Interactive music relies on real-time human expression.<n>Modern generative music AI is largely absent from this domain.<n>We propose a framework that bridges this gap.
• Score: 1.3115273020229667
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Interactive music and live performance relies on real-time human expression, but modern generative music AI remains largely absent from this domain due to its prohibitive inference latency and offline rendering paradigm. To provide pioneer musicians with a novel medium for interactive composition, we should fundamentally change these static models into dynamic, playable instruments. In this paper, we propose a framework that bridges this gap. To achieve the low latency required for live interaction without sacrificing structural coherence, we formulate distillation within a streaming autoregressive latent space. Our approach gets rid of the need for expensive paired audio-latent datasets by utilizing prompt-only inputs to synthesize teacher-guided, chunk-wise trajectories on the fly. Because live instruments require high acoustic fidelity, we introduce music-aware consistency objectives, which combine latent, spectral, and temporal-difference losses, to preserve crucial qualities like timbre, transients, and rhythmic stability during accelerated single-step streaming generation. Implemented via parameter-efficient adaptation, our distillation reduces generation steps to achieve a low real-time factor. Crucially, by operating as a continuous autoregressive stream, the system can seamlessly assimilate dynamic human inputs on the fly, allowing users to instantly steer the musical trajectory without interrupting the audio flow. Ultimately, this work recontextualizes generative text-to-music models not as passive prompt-and-wait systems, but as responsive instruments, opening new frontiers for live human-AI musical co-creation.

Related papers

• LK Jam: System Architecture and Implementation of a Real-Time Human-AI Interactive Music Generation System using Role-Aware GRU [5.407906676682284]
  LK_Jam is a real-time, bidirectional human-computer interactive music generation system.<n>It is based on a lightweight Gated Recurrent Unit (GRU) and a high-performance audio host architecture.
  arXiv  Detail & Related papers  (2026-06-19T01:03:44Z)
• Co-policy: Responsive Human-Robot Co-Creation for Musical Performances [58.92814200971293]
  Co-policy is a framework for human-robot musical co-creation.<n>It separates semantic intent grounding, constrained musical variation, and visuomotor execution.<n>We show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines.
  arXiv  Detail & Related papers  (2026-06-18T08:08:16Z)
• DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing [27.626489251186968]
  We introduce DiscoForcing, a streaming audio-driven diffusion framework.<n>It captures rhythmic structure and phase dynamics with a diffusion-forcing sequence model trained under heterogeneous noise levels.<n>It delivers more stable long-horizon rollouts and sharper audio-motion alignment than prior baselines under matched causality and latency constraints.
  arXiv  Detail & Related papers  (2026-05-27T13:47:40Z)
• EchoAvatar: Real-time Generative Avatar Animation from Audio Streams [31.328378976492775]
  We introduce a novel framework designed to generate continuous, coherent full-body motion from streaming speech and music with low latency.<n>By combining this controllability with stream audio-driven synthesis, our framework serves as a plug-and-play solution for transforming voice agents into interactive humanoid avatars.
  arXiv  Detail & Related papers  (2026-05-27T10:18:16Z)
• NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control [59.6128550986024]
  NarraScore is a hierarchical framework predicated on the core insight that emotion serves as a high-density compression of narrative logic.<n>NarraScore employs a Dual-Branch Injection strategy to reconcile global structure with local dynamism.<n>NarraScore achieves state-of-the-art consistency and narrative alignment with negligible computational overhead.
  arXiv  Detail & Related papers  (2026-02-09T09:39:42Z)
• Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length [57.458450695137664]
  We present Live Avatar, an algorithm-system co-designed framework for efficient, high-fidelity, and infinite-length avatar generation.<n>Live Avatar is first to achieve practical, real-time, high-fidelity avatar generation at this scale.
  arXiv  Detail & Related papers  (2025-12-04T11:11:24Z)
• The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity [59.78509280246215]
  Aria-Duet is an interactive system facilitating a real-time musical duet between a human pianist and Aria, a state-of-the-art generative model.<n>We analyze the system's output from a musicological perspective, finding the model can maintain stylistic semantics and develop coherent phrasal ideas.
  arXiv  Detail & Related papers  (2025-11-03T15:26:01Z)
• SAMUeL: Efficient Vocal-Conditioned Music Generation via Soft Alignment Attention and Latent Diffusion [0.0]
  We present a lightweight latent diffusion model for vocal-conditioned musical accompaniment generation.<n>The model achieves a 220 times parameter reduction compared to state-of-the-art systems while delivering 52 times faster inference.<n>The ultra-lightweight architecture enables real-time deployment on consumer hardware, making AI-assisted music creation accessible for interactive applications and resource-constrained environments.
  arXiv  Detail & Related papers  (2025-07-26T16:00:26Z)
• Extending Visual Dynamics for Video-to-Music Generation [51.274561293909926]
  DyViM is a novel framework to enhance dynamics modeling for video-to-music generation.<n>High-level semantics are conveyed through a cross-attention mechanism.<n>Experiments demonstrate DyViM's superiority over state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-04-10T09:47:26Z)
• ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformers [53.63950017886757]
  We introduce ReaLJam, an interface and protocol for live musical jamming sessions between a human and a Transformer-based AI agent trained with reinforcement learning.<n>We enable real-time interactions using the concept of anticipation, where the agent continually predicts how the performance will unfold and visually conveys its plan to the user.
  arXiv  Detail & Related papers  (2025-02-28T17:42:58Z)
• Flat latent manifolds for music improvisation between human and machine [9.571383193449648]
  We consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal improvisation is to lead to new experiences. In the learned model, we generate novel musical sequences by quantification in latent space. We provide empirical evidence for our method via a set of experiments on music and we deploy our model for an interactive jam session with a professional drummer.
  arXiv  Detail & Related papers  (2022-02-23T09:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.