Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models

• URL: http://arxiv.org/abs/2608.05126v1
• Date: Wed, 05 Aug 2026 17:50:31 GMT
• Title: Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models
• Abstract summary: Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems.<n>This work proposes Spoken Language Function Calling (SFC) to evolve beyond traditional closed-set SLU.
• Score: 14.282205541707699
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems and a pivotal link in achieving seamless human-agent interaction. While traditional SLU can effectively extract user semantics for closed-set tasks after in-domain supervised fine-tuning, it faces significant challenges in leveraging in-context learning for open-domain tasks due to its ambiguous rule definitions. This work proposes Spoken Function Calling (SFC), a novel semantic understanding perspective that optimizes semantic understanding with structured rule definitions, to evolve beyond traditional closed-set SLU. Specifically, we curate and extend a suite of spoken functions based on traditional SLU datasets, construct a multi-agent system to synthesize the SFC-Bench dataset, evaluate the performance of Large Language Models (LLMs) and Large Audio Language Models (LALMs), and enhance the SFC capabilities of LALMs through post-training. Experiments demonstrate that SFC outperforms traditional SLU, substantially enhancing the semantic extraction accuracy for LLMs and LALMs.

Related papers

• FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning [6.263790020235786]
  Foundations for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks.<n>Existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs.<n>We propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning.
  arXiv  Detail & Related papers  (2026-03-01T14:33:08Z)
• Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models [53.06230963851451]
  JARVIS is a JEPA-inspired framework for self-supervised visual enhancement in MLLMs.<n>We introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs.
  arXiv  Detail & Related papers  (2025-12-17T19:01:34Z)
• MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark [44.95801048019833]
  Spoken Language Understanding (SLU) aims to extract user semantics to execute downstream tasks.<n>There is an absence of a unified benchmark for the latest Large Language Models (LLMs) and Large Audio Language Models (LALMs)<n>This work introduces MAC-SLU, a novel Multi-Intent Automotive Cabin Spoken Language Understanding dataset.
  arXiv  Detail & Related papers  (2025-12-01T12:23:19Z)
• LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretation [5.579029325265822]
  This paper explores a hybrid approach that integrates the broad-coverage language processing of LLMs with the symbolic NLU capabilities of producing structured relational representations.<n>Our results suggest that our hybrid method works significantly better than the symbolic-only pipeline.
  arXiv  Detail & Related papers  (2025-10-22T19:38:20Z)
• Uni-Sign: Toward Unified Sign Language Understanding at Scale [90.76641997060513]
  We propose a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks.<n>Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks.
  arXiv  Detail & Related papers  (2025-01-25T11:51:23Z)
• Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series [3.453940014682793]
  We propose Context-Alignment to align time series (TS) data with a linguistic component in language environments familiar to Large Language Models (LLMs)<n>Such context-level alignment comprises structural alignment and logical alignment, which is achieved by a Dual-Scale Context-Alignment GNNs (DSCA-GNNs)<n>Extensive experiments show the effectiveness of DECA and the importance of Context-Alignment across tasks, particularly in few-shot and zero-shot forecasting.
  arXiv  Detail & Related papers  (2025-01-07T12:40:35Z)
• LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation [72.02635550088546]
  This work explores how large language models (LLMs) can enhance CLIP's capability, especially for processing longer and more complex image captions.<n>We introduce a caption-to-caption contrastive fine-tuning framework, significantly enhancing the discriminative quality of LLM outputs.<n>Our approach outperforms LoRA-based methods, achieving nearly fourfold faster training with superior performance.
  arXiv  Detail & Related papers  (2024-11-07T18:59:16Z)
• SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization [70.11167263638562]
  Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. We first present a simple yet well-crafted framework named name, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework.
  arXiv  Detail & Related papers  (2024-10-28T18:10:26Z)
• Large Language Models are Interpretable Learners [53.56735770834617]
  In this paper, we show a combination of Large Language Models (LLMs) and symbolic programs can bridge the gap between expressiveness and interpretability. The pretrained LLM with natural language prompts provides a massive set of interpretable modules that can transform raw input into natural language concepts. As the knowledge learned by LSP is a combination of natural language descriptions and symbolic rules, it is easily transferable to humans (interpretable) and other LLMs.
  arXiv  Detail & Related papers  (2024-06-25T02:18:15Z)
• ECLM: Entity Level Language Model for Spoken Language Understanding with Chain of Intent [20.3491383818583]
  Large Language Models (LLMs) have demonstrated impressive capabilities in language generation and general task performance.<n>We propose the Entity-level Language Model (ECLM) framework, which reformulates slot-filling as an entity recognition task.<n>ECLM significantly outperforms strong baselines such as Uni-MIS, achieving gains of 3.7% on MixATIS and 3.1% on MixSNIPS.
  arXiv  Detail & Related papers  (2024-03-07T13:30:52Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.