Recent Advances in Discrete Speech Tokens: A Review

• URL: http://arxiv.org/abs/2502.06490v2
• Date: Sun, 16 Feb 2025 08:11:09 GMT
• Title: Recent Advances in Discrete Speech Tokens: A Review
• Authors: Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, Kai Yu,
• Abstract summary: discrete speech tokens, characterized by their discrete, compact, and concise nature, are advantageous for efficient transmission and storage. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types.
• Score: 25.038527125935747
• License:
• Abstract: The rapid advancement of speech generation technologies in the era of large language models (LLMs) has established discrete speech tokens as a foundational paradigm for speech representation. These tokens, characterized by their discrete, compact, and concise nature, are not only advantageous for efficient transmission and storage, but also inherently compatible with the language modeling framework, enabling seamless integration of speech into text-dominated LLM architectures. Current research categorizes discrete speech tokens into two principal classes: acoustic tokens and semantic tokens, each of which has evolved into a rich research domain characterized by unique design philosophies and methodological approaches. This survey systematically synthesizes the existing taxonomy and recent innovations in discrete speech tokenization, conducts a critical examination of the strengths and limitations of each paradigm, and presents systematic experimental comparisons across token types. Furthermore, we identify persistent challenges in the field and propose potential research directions, aiming to offer actionable insights to inspire future advancements in the development and application of discrete speech tokens.

Related papers

• A Comparative Study of Discrete Speech Tokens for Semantic-Related Tasks with Large Language Models [46.298114175792584]
  We present a fair and thorough comparison between discrete and continuous features across a variety of semantic-related tasks. Our findings reveal that continuous features generally outperform discrete tokens, particularly in tasks requiring fine-grained semantic understanding.
  arXiv  Detail & Related papers  (2024-11-13T16:20:20Z)
• Roadmap towards Superhuman Speech Understanding using Large Language Models [60.57947401837938]
  Large language models (LLMs) integrate speech and audio data. Recent advances, such as GPT-4o, highlight the potential for end-to-end speech LLMs. We propose a five-level roadmap, ranging from basic automatic speech recognition (ASR) to advanced superhuman models.
  arXiv  Detail & Related papers  (2024-10-17T06:44:06Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• dMel: Speech Tokenization made Simple [19.169460770473908]
  We show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation (dMel) Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework.
  arXiv  Detail & Related papers  (2024-07-22T17:51:53Z)
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment [82.86363991170546]
  We propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities. Our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. These findings highlight the potential to reshape instruction-following SLMs by incorporating descriptive rich, speech captions.
  arXiv  Detail & Related papers  (2024-06-27T03:52:35Z)
• DASB -- Discrete Audio and Speech Benchmark [12.02056212008393]
  We release the Discrete Audio and Speech Benchmark (DASB), a leaderboard for benchmarking discrete audio tokens across a range of tasks. Our results show that, on average, semantic tokens outperform compression tokens across most discriminative and generative tasks. However, the performance gap between semantic tokens and standard continuous representations remains substantial.
  arXiv  Detail & Related papers  (2024-06-20T13:23:27Z)
• Non-verbal information in spontaneous speech -- towards a new framework of analysis [0.5559722082623594]
  This paper offers an analytical schema and a technological proof-of-concept for the categorization of prosodic signals. We present a classification process that disentangles prosodic phenomena of three orders. Disentangling prosodic patterns can direct a theory of communication and speech organization.
  arXiv  Detail & Related papers  (2024-03-06T08:03:05Z)
• Revisiting Conversation Discourse for Dialogue Disentanglement [88.3386821205896]
  We propose enhancing dialogue disentanglement by taking full advantage of the dialogue discourse characteristics. We develop a structure-aware framework to integrate the rich structural features for better modeling the conversational semantic context. Our work has great potential to facilitate broader multi-party multi-thread dialogue applications.
  arXiv  Detail & Related papers  (2023-06-06T19:17:47Z)
• BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models [56.93604813379634]
  Self-supervised techniques for learning speech representations have been shown to develop linguistic competence from exposure to speech without the need for human labels. We propose a language-acquisition-friendly benchmark to probe spoken language models at the lexical and syntactic levels. We highlight two exciting challenges that need to be addressed for further progress: bridging the gap between text and speech and between clean speech and in-the-wild speech.
  arXiv  Detail & Related papers  (2023-06-02T12:54:38Z)
• Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis [3.691712391306624]
  We show that the fine-grained latent space also captures coarse-grained information, which is more evident as the dimension of latent space increases in order to capture diverse prosodic representations. We alleviate this issue by first capturing rich speech attributes into a token-level latent space and then, separately train a prior network that given the input text, learns utterance-level representations in order to predict the phoneme-level, posterior latents extracted during the previous step.
  arXiv  Detail & Related papers  (2022-11-01T15:17:25Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.