On The Landscape of Spoken Language Models: A Comprehensive Survey

• URL: http://arxiv.org/abs/2504.08528v1
• Date: Fri, 11 Apr 2025 13:40:53 GMT
• Title: On The Landscape of Spoken Language Models: A Comprehensive Survey
• Authors: Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, Shinji Watanabe,
• Abstract summary: spoken language models (SLMs) act as universal speech processing systems.<n>Work in this area is very diverse, with a range of terminology and evaluation settings.
• Score: 144.11278973534203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems. This trend is similar to the progression toward universal language models that has taken place in the field of (text) natural language processing. SLMs include both "pure" language models of speech -- models of the distribution of tokenized speech sequences -- and models that combine speech encoders with text language models, often including both spoken and written input or output. Work in this area is very diverse, with a range of terminology and evaluation settings. This paper aims to contribute an improved understanding of SLMs via a unifying literature survey of recent work in the context of the evolution of the field. Our survey categorizes the work in this area by model architecture, training, and evaluation choices, and describes some key challenges and directions for future work.

Related papers

• Linguistic Interpretability of Transformer-based Language Models: a systematic review [1.3194391758295114]
  Language models based on the Transformer architecture achieve excellent results in many language-related tasks. However, little is known about how their internal computations help them achieve their results. There is, however, a line of research -- 'interpretability' -- aiming to learn how information is encoded inside these models.
  arXiv  Detail & Related papers  (2025-04-09T08:00:12Z)
• LAST: Language Model Aware Speech Tokenization [24.185165710384997]
  We propose a novel approach to training a speech tokenizer by leveraging objectives from pre-trained textual LMs. Our aim is to transform features from a pre-trained speech model into a new feature space that enables better clustering for speech LMs.
  arXiv  Detail & Related papers  (2024-09-05T16:57:39Z)
• Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics [4.9639158834745745]
  We introduce an innovative masking approach leveraging genre and topicality information to tailor language models to specialized domains. Our method incorporates a ranking process that prioritizes words based on their significance, subsequently guiding the masking procedure. Experiments conducted using continual pre-training within the legal domain have underscored the efficacy of our approach on the LegalGLUE benchmark in the English language.
  arXiv  Detail & Related papers  (2024-02-19T10:43:27Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets.<n>This survey delves into an important attribute of these datasets: the dialect of a language.<n>Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• An Overview on Language Models: Recent Developments and Outlook [32.528770408502396]
  Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner. Pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications.
  arXiv  Detail & Related papers  (2023-03-10T07:55:00Z)
• PaLM-E: An Embodied Multimodal Language Model [101.29116156731762]
  We propose embodied language models to incorporate real-world continuous sensor modalities into language models. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks. Our largest model, PaLM-E-562B with 562B parameters, is a visual-language generalist with state-of-the-art performance on OK-VQA.
  arXiv  Detail & Related papers  (2023-03-06T18:58:06Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Specializing Multilingual Language Models: An Empirical Study [50.7526245872855]
  Contextualized word representations from pretrained multilingual language models have become the de facto standard for addressing natural language tasks. For languages rarely or never seen by these models, directly using such models often results in suboptimal representation or use of data.
  arXiv  Detail & Related papers  (2021-06-16T18:13:55Z)
• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques [15.906959137350247]
  This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work.
  arXiv  Detail & Related papers  (2021-04-27T14:32:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.