Exploring Token-Space Manipulation in Latent Audio Tokenizers

• URL: http://arxiv.org/abs/2605.11192v1
• Date: Mon, 11 May 2026 19:58:14 GMT
• Title: Exploring Token-Space Manipulation in Latent Audio Tokenizers
• Abstract summary: We propose the Latent Audio Tokenizer for Token-space Editing (LATTE)<n>LATTE appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding.<n>We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings.
• Score: 33.022035588157614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural audio codecs provide compact discrete representations for speech generation and manipulation. However, most codecs organize tokens as frame-level sequences, making it difficult to study or intervene on global factors of variation. In this work, we propose the Latent Audio Tokenizer for Token-space Editing (LATTE) that appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding. This design produces a compact, non-temporally aligned bottleneck in which each token can aggregate global information across the full utterance. We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings while enabling simple token-space interventions. In particular, we find that swapping selected latent token positions between utterances can modify global attributes, such as speaker identity and background noise, and we evaluate these interventions on voice conversion and denoising tasks. Our results suggest that compact latent audio tokenizers can support controllable audio manipulation without supervision in task-specific editing models.

Related papers

• Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition [58.25449304752214]
  We study whether explicit acoustic cues are grounded when the raw audio is already available.<n>We deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set.<n>Aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance.<n>We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
  arXiv  Detail & Related papers  (2026-06-05T14:26:06Z)
• PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization [8.126475706520063]
  PairAlign is a framework for compact audio tokenization through sequence-level self-alignment.<n>It learns compact, non-degenerate sequences with broad vocabulary usage.<n>It preserves edit-distance search while reducing archive token count by 55%.
  arXiv  Detail & Related papers  (2026-05-07T17:11:22Z)
• Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization [27.32235541083431]
  We introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer.<n>DyCAST learns to associate tokens with character-level linguistic units during training.<n>We also introduce a retrieval-augmented decoding mechanism that enhances reconstruction without increasing fidelity.
  arXiv  Detail & Related papers  (2026-01-30T16:58:40Z)
• AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation [16.047087043580053]
  Multimodal Large Language Models (MLLMs) have been widely applied in speech and music.<n>Unlike semantic-only text tokens, audio tokens must both capture global semantic content and preserve fine-grained acoustic details.<n>This paper provides suitable definitions for semantic and acoustic tokens and introduces a systematic evaluation framework.
  arXiv  Detail & Related papers  (2025-09-02T14:15:22Z)
• Discrete Audio Tokens: More Than a Survey! [137.3721175670642]
  This paper presents a systematic review and benchmark of discrete audio tokenizers.<n>It covers speech, music, and general audio domains.<n>We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains.
  arXiv  Detail & Related papers  (2025-06-12T01:35:43Z)
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens [49.569695524535454]
  We propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
  arXiv  Detail & Related papers  (2024-07-07T15:16:19Z)
• How Should We Extract Discrete Audio Tokens from Self-Supervised Models? [15.03039528965825]
  This paper explores the optimal configuration of semantic tokens across discriminative and generative tasks. We propose a scalable solution to train a universal vocoder across multiple SSL layers.
  arXiv  Detail & Related papers  (2024-06-15T20:43:07Z)
• Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models [98.34889301515412]
  We develop the Qwen-Audio model and address the limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types. Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning. We further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
  arXiv  Detail & Related papers  (2023-11-14T05:34:50Z)
• TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition [51.565319173790314]
  TokenSplit is a sequence-to-sequence encoder-decoder model that uses the Transformer architecture. We show that our model achieves excellent performance in terms of separation, both with or without transcript conditioning. We also measure the automatic speech recognition (ASR) performance and provide audio samples of speech synthesis to demonstrate the additional utility of our model.
  arXiv  Detail & Related papers  (2023-08-21T01:52:01Z)
• Controllable speech synthesis by learning discrete phoneme-level prosodic representations [53.926969174260705]
  We present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset.
  arXiv  Detail & Related papers  (2022-11-29T15:43:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.