Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks

• URL: http://arxiv.org/abs/2310.12477v2
• Date: Sat, 15 Jun 2024 14:13:54 GMT
• Title: Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Tasks
• Authors: Ming-Hao Hsu, Kai-Wei Chang, Shang-Wen Li, Hung-yi Lee,
• Abstract summary: In-context learning (ICL) has played an essential role in utilizing large language models (LLMs) This study is the first work exploring ICL for speech classification tasks with textless speech LM.
• Score: 98.5311231450689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ever since the development of GPT-3 in the natural language processing (NLP) field, in-context learning (ICL) has played an essential role in utilizing large language models (LLMs). By presenting the LM utterance-label demonstrations at the input, the LM can accomplish few-shot learning without relying on gradient descent or requiring explicit modification of its parameters. This enables the LM to perform various downstream tasks in a black-box manner. Despite the success of ICL in NLP, little work is exploring the possibility of ICL in speech processing. This study is the first work exploring ICL for speech classification tasks with textless speech LM. We first show that the current speech LM lacks the ICL capability. We then perform warmup training on the speech LM, equipping the LM with demonstration learning capability. This paper explores and proposes the first speech LM capable of performing unseen classification tasks in an ICL manner.

Related papers

• In-Context Learning Enables Robot Action Prediction in LLMs [52.285739178561705]
  We introduce RoboPrompt, a framework that enables offthe-shelf text-only Large Language Models to directly predict robot actions. Our approach firstally identifiess that capture important moments from an episode. We extract end-effector actions as well as the estimated initial object poses, and both are converted into textual descriptions. This enables an LLM to directly predict robot actions at test time.
  arXiv  Detail & Related papers  (2024-10-16T17:56:49Z)
• Self-Powered LLM Modality Expansion for Large Speech-Text Models [62.27700381806554]
  Large language models (LLMs) exhibit remarkable performance across diverse tasks. This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning. We introduce a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning.
  arXiv  Detail & Related papers  (2024-10-04T04:34:24Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• What Do Language Models Learn in Context? The Structured Task Hypothesis [89.65045443150889]
  Large language models (LLMs) learn a novel task from in-context examples presented in a demonstration, termed in-context learning (ICL) One popular hypothesis explains ICL by task selection. Another popular hypothesis is that ICL is a form of meta-learning, i.e., the models learn a learning algorithm at pre-training time and apply it to the demonstration.
  arXiv  Detail & Related papers  (2024-06-06T16:15:34Z)
• BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing [35.31866559807704]
  modality alignment between speech and text remains an open problem. We propose the BLSP approach that bootstraps Language-Speech Pre-training via behavior alignment of continuation writing. We demonstrate that this straightforward process can extend the capabilities of LLMs to speech, enabling speech recognition, speech translation, spoken language understanding, and speech conversation, even in zero-shot cross-lingual scenarios.
  arXiv  Detail & Related papers  (2023-09-02T11:46:05Z)
• Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study [0.0]
  This paper explores the integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems. Our primary focus is to investigate the potential of using an LLM's in-context learning capabilities to enhance the performance of ASR systems.
  arXiv  Detail & Related papers  (2023-07-13T02:31:55Z)
• Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models [7.782346535009883]
  This work introduces approaches to assessing phrase breaks in ESL learners' speech using pre-trained language models (PLMs) and large language models (LLMs)
  arXiv  Detail & Related papers  (2023-06-08T07:10:39Z)
• What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning [24.395288160951118]
  Large language models (LLMs) exploit in-context learning (ICL) to solve tasks with only a few demonstrations. We characterize two ways through which ICL leverages demonstrations. We show that models can achieve non-trivial performance with only TR, and TR does not further improve with larger models or more demonstrations.
  arXiv  Detail & Related papers  (2023-05-16T18:05:19Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.