MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark

• URL: http://arxiv.org/abs/2512.01603v1
• Date: Mon, 01 Dec 2025 12:23:19 GMT
• Title: MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark
• Authors: Yuezhang Peng, Chonghao Cai, Ziang Liu, Shuai Fan, Sheng Jiang, Hua Xu, Yuxin Liu, Qiguang Chen, Kele Xu, Yao Li, Sheng Wang, Libo Qin, Xie Chen,
• Abstract summary: Spoken Language Understanding (SLU) aims to extract user semantics to execute downstream tasks.<n>There is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs)<n>This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding dataset.
• Score: 44.95801048019833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spoken Language Understanding (SLU), which aims to extract user semantics to execute downstream tasks, is a crucial component of task-oriented dialog systems. Existing SLU datasets generally lack sufficient diversity and complexity, and there is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs). This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding Dataset, which increases the difficulty of the SLU task by incorporating authentic and complex multi-intent data. Based on MAC-SLU, we conducted a comprehensive benchmark of leading open-source LLMs and LALMs, covering methods like in-context learning, supervised fine-tuning (SFT), and end-to-end (E2E) and pipeline paradigms. Our experiments show that while LLMs and LALMs have the potential to complete SLU tasks through in-context learning, their performance still lags significantly behind SFT. Meanwhile, E2E LALMs demonstrate performance comparable to pipeline approaches and effectively avoid error propagation from speech recognition. Code\footnote{https://github.com/Gatsby-web/MAC\_SLU} and datasets\footnote{huggingface.co/datasets/Gatsby1984/MAC\_SLU} are released publicly.

Related papers

• UniSLU: Unified Spoken Language Understanding from Heterogeneous Cross-Task Datasets [21.47194295019577]
  Spoken Language Understanding (SLU) plays a crucial role in speech-centric multimedia applications.<n>We propose UniSLU, a unified framework that jointly models multiple SLU tasks within a single architecture.
  arXiv  Detail & Related papers  (2025-07-17T09:45:49Z)
• UCSC at SemEval-2025 Task 3: Context, Models and Prompt Optimization for Automated Hallucination Detection in LLM Output [7.121378498209948]
  SemEval 2025 Task 3, Mu-SHROOM: Shared-task on Hallucinations and Related Observable Overgeneration Mistakes, is a recent effort in this direction.<n>This paper describes the UCSC system submission to the shared Mu-SHROOM task.<n>We introduce a framework that first retrieves relevant context, next identifies false content from the answer, and finally maps them back to spans in the LLM output.
  arXiv  Detail & Related papers  (2025-05-05T21:15:40Z)
• Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions [60.43398881149664]
  We introduce LOS-Net, a lightweight attention-based architecture trained on an efficient encoding of the LLM Output Signature.<n>It achieves superior performance across diverse benchmarks and LLMs, while maintaining extremely low detection latency.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration [10.970637831760136]
  Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality.<n>We introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference.<n>Our experiments demonstrate that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text.
  arXiv  Detail & Related papers  (2024-10-09T14:15:30Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Large Language Models for Expansion of Spoken Language Understanding Systems to New Languages [0.20971479389679337]
  Spoken Language Understanding (SLU) models are a core component of voice assistants (VA), such as Alexa, Bixby, and Google Assistant. In this paper, we introduce a pipeline designed to extend SLU systems to new languages, utilizing Large Language Models (LLMs) Our approach improved on the MultiATIS++ benchmark, a primary multi-language SLU dataset, in the cloud scenario using an mBERT model.
  arXiv  Detail & Related papers  (2024-04-03T09:13:26Z)
• ECLM: Entity Level Language Model for Spoken Language Understanding with Chain of Intent [20.3491383818583]
  Large Language Models (LLMs) have demonstrated impressive capabilities in language generation and general task performance.<n>We propose the Entity-level Language Model (ECLM) framework, which reformulates slot-filling as an entity recognition task.<n>ECLM significantly outperforms strong baselines such as Uni-MIS, achieving gains of 3.7% on MixATIS and 3.1% on MixSNIPS.
  arXiv  Detail & Related papers  (2024-03-07T13:30:52Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• InfMLLM: A Unified Framework for Visual-Language Tasks [44.29407348046122]
  multimodal large language models (MLLMs) have attracted growing interest. This work delves into enabling LLMs to tackle more vision-language-related tasks. InfMLLM achieves either state-of-the-art (SOTA) performance or performance comparable to recent MLLMs.
  arXiv  Detail & Related papers  (2023-11-12T09:58:16Z)
• STOP: A dataset for Spoken Task Oriented Semantic Parsing [66.14615249745448]
  End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. We release the Spoken Task-Oriented semantic Parsing (STOP) dataset, the largest and most complex SLU dataset to be publicly available. In addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
  arXiv  Detail & Related papers  (2022-06-29T00:36:34Z)
• ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet [95.39817519115394]
  ESPnet-SLU is a project inside end-to-end speech processing toolkit, ESPnet. It is designed for quick development of spoken language understanding in a single framework.
  arXiv  Detail & Related papers  (2021-11-29T17:05:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.