Communicating Sound Through Natural Language

• URL: http://arxiv.org/abs/2605.08750v1
• Date: Sat, 09 May 2026 07:25:54 GMT
• Title: Communicating Sound Through Natural Language
• Abstract summary: We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language.<n>Under fixed system prompts, the agents write their own analysis and synthesis code, communicating only through a lexical sentence.<n>We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity.
• Score: 20.119057003948004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural language is widely used to describe, prompt, and control audio systems, but rarely serves as the representation carrying audio itself. We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language. Under fixed system prompts, the agents write their own analysis and synthesis code, communicating only through a lexical sentence, shared vocabulary, and optional symbolic music structure. The sender analyzes an input waveform into interpretable, non-learned acoustic descriptors, quantizes each with a feature-specific interval vocabulary, and verbalizes the lexical code as English. The receiver parses the sentence back into lexical-acoustic constraints and renders a waveform through closed-loop refinement. The transmitted text serves as both a rich caption and as the transport representation itself. We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity. Experiments on short sounds and symbolic music transfer show that plain text preserves measurable acoustic structure while remaining interpretable, editable, and native to LLM-mediated communication.

Related papers

• Unified Audio Intelligence Without Regressing on Text Intelligence [86.55418188552768]
  We introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM.<n>Audex adopts a simple unified design with a single Transformer decoder.<n>Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation.
  arXiv  Detail & Related papers  (2026-07-06T15:11:57Z)
• Libretto: Giving LLM Agents a Sense of Musical Structure [4.144744763257738]
  We introduce Libretto, a framework for symbolic music generation and revision.<n>Libretto evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation.<n>Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
  arXiv  Detail & Related papers  (2026-06-21T22:56:07Z)
• Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts [20.986457042343684]
  We introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation.<n>PlanAudio is a unified, autoregressive LLM-based framework.<n>We perform evaluations in the scenarios of speech, sound, and their composites.
  arXiv  Detail & Related papers  (2026-05-27T07:15:35Z)
• Beyond Transcripts: A Renewed Perspective on Audio Chaptering [66.61445564139052]
  We show that a novel audio-only architecture (AudioSeg) outperforms text-based approaches for segmenting long-form audio into coherent sections.<n>Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and MLLMs remain limited by context length and weak instruction following.
  arXiv  Detail & Related papers  (2026-02-09T18:28:10Z)
• Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics? [10.420650470020844]
  We evaluate the above three joint language-audio embedding models on their ability to capture perceptual dimensions of timbre.<n>Our findings show that LAION-CLAP consistently provides the most reliable alignment with human-perceived timbre semantics across both instrumental sounds and audio effects.
  arXiv  Detail & Related papers  (2025-10-16T03:01:41Z)
• SeeingSounds: Learning Audio-to-Visual Alignment via Text [15.011814561603964]
  We introduce SeeingSounds, a framework for audio-to-image generation that leverages the interplay between audio, language, and vision.<n>Our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model.<n>This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception.
  arXiv  Detail & Related papers  (2025-10-10T18:42:50Z)
• Text-Queried Audio Source Separation via Hierarchical Modeling [53.94434504259829]
  We propose a hierarchical decomposition framework, HSM-TSS, that decouples the task into global-local semantic-guided feature separation and structure-preserving acoustic reconstruction.<n>A Q-Audio architecture is employed to align audio and text modalities, serving as pretrained global-semantic encoders.<n>Our method achieves state-of-the-art separation performance with data-efficient training while maintaining superior semantic consistency with queries in complex auditory scenes.
  arXiv  Detail & Related papers  (2025-05-27T11:00:38Z)
• Steganography Beyond Space-Time with Chain of Multimodal AI [8.095373104009868]
  Steganography is the art and science of covert writing.<n>As artificial intelligence continues to evolve, its ability to synthesise realistic content emerges as a threat in the hands of cybercriminals.<n>This study proposes a paradigm in steganography for audiovisual media, where messages are concealed beyond both spatial and temporal domains.
  arXiv  Detail & Related papers  (2025-02-25T15:56:09Z)
• ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Sounds [6.751004034983776]
  We introduce ISPA (Inter-Species Phonetic Alphabet), a precise, concise, and interpretable system for transcribing animal sounds into text. We show that established human language ML paradigms and models, such as language models, can be successfully applied to improve performance.
  arXiv  Detail & Related papers  (2024-02-05T18:27:27Z)
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT [65.69648099999439]
  Generative Pre-trained Transformer (GPT) models have achieved remarkable performance on various natural language processing tasks. We propose LauraGPT, a novel unified audio-and-text GPT-based LLM for audio recognition, understanding, and generation.
  arXiv  Detail & Related papers  (2023-10-07T03:17:59Z)
• On decoder-only architecture for speech-to-text and large language model integration [59.49886892602309]
  Speech-LLaMA is a novel approach that effectively incorporates acoustic information into text-based large language models. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2023-07-08T06:47:58Z)
• A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units [94.64927912924087]
  Existing systems ignore the correlation between prosody and language content, leading to degradation of naturalness in converted speech. We devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability.
  arXiv  Detail & Related papers  (2022-11-12T00:54:09Z)
• Vector-Quantized Timbre Representation [53.828476137089325]
  This paper targets a more flexible synthesis of an individual timbre by learning an approximate decomposition of its spectral properties with a set of generative features. We introduce an auto-encoder with a discrete latent space that is disentangled from loudness in order to learn a quantized representation of a given timbre distribution. We detail results for translating audio between orchestral instruments and singing voice, as well as transfers from vocal imitations to instruments.
  arXiv  Detail & Related papers  (2020-07-13T12:35:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.