A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models

• URL: http://arxiv.org/abs/2607.00309v1
• Date: Wed, 01 Jul 2026 01:21:42 GMT
• Title: A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models
• Abstract summary: We present a real-time musical interface that converts natural-language scene descriptions into evolving procedural soundscapes.<n>A performer types a prompt such as "warm jazz cafe at midnight" and steers it through direct parameter adjustments.<n>We evaluate text-audio semantic alignment using LAION-CLAP on held-out prompts as a technical proxy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a real-time musical interface that converts natural-language scene descriptions into evolving procedural soundscapes. A performer types a prompt such as "warm jazz cafe at midnight" and steers it through direct parameter adjustments - stepping brightness down, switching a rhythm style - each producing a predictable, audible shift without re-prompting. Where GPU-bound text-to-audio systems synthesize monolithic waveforms, our instrument generates human-readable configurations over a categorical schema, enabling fine-grained performer control; most valid combinations are designed to sound musically coherent. Three interchangeable backends - embedding retrieval for sub-second CPU-only use, hosted LLMs via API, and a fine-tuned 270M local model - all emit the same schema. A live generator architecture continuously emits audio while resolving new instructions in the background, crossfading seamlessly when ready; even when an LLM takes 5-12 seconds to respond, the audience hears uninterrupted sound - reframing text-to-music as an ongoing performable stream rather than a one-shot generation. We evaluate text-audio semantic alignment using LAION-CLAP on held-out prompts as a technical proxy, finding that retrieval-based configuration outperforms random valid configurations on this metric, while noting that LAION-CLAP also informed retrieval-map construction. We report performance observations, informal listener feedback, and release materials for the SDK, dataset artifacts, model, and audiovisual performance interface.

Related papers

• BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation [1.8763872698583384]
  We present BackgroundMellow, a framework that treats story-to-audio generation as a precise orchestration and signal processing problem.<n>This framework is enabled without ground-truth through a master-specialist agent architecture that decomposes text into precise and multi-layered audio cues.<n>Our pipeline is built over Tango2 latent diffusion model for environmental synthesis and a novel Cinematic BGM Retriever mined from professional soundtracks.
  arXiv  Detail & Related papers  (2026-07-13T10:28:15Z)
• Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors [10.603967327834214]
  Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision.<n>Our method, ScenA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data.<n>We evaluate ScenA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics.
  arXiv  Detail & Related papers  (2026-06-17T17:51:50Z)
• Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models [57.635707525420884]
  We propose a framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features.<n>We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation.
  arXiv  Detail & Related papers  (2026-06-14T11:23:09Z)
• Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources [57.68713138159972]
  We present a time-resolved audio encoder that learns event semantics together with source trajectories.<n>Experiments show that this representation improves the semanticlocalization and yields stronger reasoning than spatial and localization-oriented tradeoffs.
  arXiv  Detail & Related papers  (2026-06-12T05:58:31Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts [20.986457042343684]
  We introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation.<n>PlanAudio is a unified, autoregressive LLM-based framework.<n>We perform evaluations in the scenarios of speech, sound, and their composites.
  arXiv  Detail & Related papers  (2026-05-27T07:15:35Z)
• InstructAudio: Unified speech and music generation with natural language instruction [52.76518112649456]
  InstructAudio is a unified framework that enables instruction-based control of acoustic attributes.<n>It supports expressive speech, music, and dialogue generation in English and Chinese.
  arXiv  Detail & Related papers  (2025-11-23T15:15:21Z)
• SeeingSounds: Learning Audio-to-Visual Alignment via Text [15.011814561603964]
  We introduce SeeingSounds, a framework for audio-to-image generation that leverages the interplay between audio, language, and vision.<n>Our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model.<n>This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception.
  arXiv  Detail & Related papers  (2025-10-10T18:42:50Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation [17.41880273107978]
  Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) We propose Contrastive Long-form Language-Audio Pretraining (textbfCoLLAP) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words) We collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds.
  arXiv  Detail & Related papers  (2024-10-03T07:46:51Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.