UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions

• URL: http://arxiv.org/abs/2310.02973v2
• Date: Wed, 3 Apr 2024 14:12:36 GMT
• Title: UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions
• Authors: Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Livescu, Shinji Watanabe,
• Abstract summary: We build a single model that jointly performs various spoken language understanding (SLU) tasks. We demonstrate the efficacy of our single multi-task learning model "UniverSLU" for 12 speech classification and sequence generation task types spanning 17 datasets and 9 languages.
• Score: 64.50935101415776
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model's behavior and surpassing performance of task-specific models. Motivated by this, we ask: can we build a single model that jointly performs various spoken language understanding (SLU) tasks? We start by adapting a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers. We enhance this approach through instruction tuning, i.e., finetuning by describing the task using natural language instructions followed by the list of label options. Our approach can generalize to new task descriptions for the seen tasks during inference, thereby enhancing its user-friendliness. We demonstrate the efficacy of our single multi-task learning model "UniverSLU" for 12 speech classification and sequence generation task types spanning 17 datasets and 9 languages. On most tasks, UniverSLU achieves competitive performance and often even surpasses task-specific models. Additionally, we assess the zero-shot capabilities, finding that the model generalizes to new datasets and languages for seen task types.

Related papers

• Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model [45.161909551392085]
  We propose finding task-specificworks within a multi-task spoken language understanding model via neural network pruning. We show that pruned models were successful in adapting to additional ASR or IC data with minimal performance degradation on previously trained tasks.
  arXiv  Detail & Related papers  (2024-06-18T06:39:41Z)
• SpeechVerse: A Large-scale Generalizable Audio Language Model [38.67969337605572]
  SpeechVerse is a robust multi-task training and curriculum learning framework. It combines pre-trained speech and text foundation models via a small set of learnable parameters. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks.
  arXiv  Detail & Related papers  (2024-05-14T03:33:31Z)
• Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts [75.75548749888029]
  We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.
  arXiv  Detail & Related papers  (2023-05-11T17:57:49Z)
• SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks [88.4408774253634]
  Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community. There are not nearly as many SLU task benchmarks, and many of the existing ones use data that is not freely available to all researchers. Recent work has begun to introduce such benchmark for several tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:39:59Z)
• Improving Task Generalization via Unified Schema Prompt [87.31158568180514]
  Unified Prompt is a flexible and prompting method, which automatically customizes the learnable prompts for each task according to the task input schema. It models the shared knowledge between tasks, while keeping the characteristics of different task schema. The framework achieves strong zero-shot and few-shot performance on 16 unseen tasks downstream from 8 task types.
  arXiv  Detail & Related papers  (2022-08-05T15:26:36Z)
• Multitask Prompted Training Enables Zero-Shot Task Generalization [70.12770442071657]
  We develop a system for mapping general natural language tasks into a human-readable prompted form. We fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models 16x its size.
  arXiv  Detail & Related papers  (2021-10-15T17:08:57Z)
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation [80.18830380517753]
  We develop a new task-agnostic distillation framework XtremeDistilTransformers. We study the transferability of several source tasks, augmentation resources and model architecture for distillation.
  arXiv  Detail & Related papers  (2021-06-08T17:49:33Z)
• Modelling Latent Skills for Multitask Language Generation [15.126163032403811]
  We present a generative model for multitask conditional language generation. Our guiding hypothesis is that a shared set of latent skills underlies many disparate language generation tasks. We instantiate this task embedding space as a latent variable in a latent variable sequence-to-sequence model.
  arXiv  Detail & Related papers  (2020-02-21T20:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.