SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies

• URL: http://arxiv.org/abs/2607.25716v1
• Date: Tue, 28 Jul 2026 13:43:04 GMT
• Title: SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies
• Abstract summary: Federated learning (FL) enables privacy-preserving training of automatic speech recognition (ASR) systems across distributed data sources.<n>This paper presents the first systematic study of federated training for SpeechLLM-based end-to-end ASR systems.
• Score: 15.883924147497817
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) enables privacy-preserving training of automatic speech recognition (ASR) systems across distributed data sources, yet its application to large-scale speech language models (SpeechLLMs) remains unexplored. This paper presents the first systematic study of federated training for SpeechLLM-based end-to-end ASR systems. We design a communication-efficient federated optimization strategy tailored to the unique challenges of SpeechLLM architectures, addressing high-dimensional parameter spaces, gradient communication overhead, and computational constraints in distributed settings. Through extensive empirical evaluation on monolingual ASR tasks in English and Italian, we demonstrate the effectiveness and stability of our federated approach compared to centralized training baselines across diverse acoustic conditions and speaking styles. Additionally, we conduct a comprehensive ablation study analyzing the impact of different speech encoder architectures on monolingual English ASR performance within the federated framework, providing insights into optimal model configurations for decentralized training. Our results achieve competitive word error rates while reducing communication costs, establishing practical foundations for federated SpeechLLM deployment in real-world multilingual scenarios.

Related papers

• LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families [53.111005425237245]
  Large language models (LLMs) have driven substantial advances in speech language models (SpeechLMs)<n>LoASR-Bench comprises 25 languages from 9 language families, featuring both Latin and non-Latin scripts.<n> Experimental results highlight the limitations of the latest SpeechLMs in handling real-world low-resource languages.
  arXiv  Detail & Related papers  (2026-03-20T15:26:34Z)
• Multimodal In-context Learning for ASR of Low-resource Languages [16.078416187950207]
  In-context learning (ICL) with large language models (LLMs) addresses this problem.<n>This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL)<n>Cross-lingual transfer learning improves MICL efficiency on target languages without training on them.
  arXiv  Detail & Related papers  (2026-01-09T10:52:23Z)
• Efficient and Versatile Model for Multilingual Information Retrieval of Islamic Text: Development and Deployment in Real-World Scenarios [1.2891210250935148]
  We leverage the unique characteristics of the Quranic multilingual corpus to examine the optimal strategies to develop an ad-hoc IR system for the Islamic domain.<n>We prepared eleven retrieval models employing four training approaches: monolingual, cross-lingual, translate-train-all, and a novel mixed method combining cross-lingual and monolingual techniques.
  arXiv  Detail & Related papers  (2025-09-18T19:32:07Z)
• MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models [59.80042864360884]
  Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately.<n>This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions.
  arXiv  Detail & Related papers  (2024-11-27T09:01:08Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios [76.11409260727459]
  This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance.
  arXiv  Detail & Related papers  (2024-06-13T08:16:52Z)
• Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets [22.29915616018026]
  Large Language Models (LLMs) have demonstrated unparalleled effectiveness in various NLP tasks. Our research aims to evaluate the impact of various configurations of speech encoders, LLMs, and projector modules. We introduce a three-stage training approach, expressly developed to enhance the model's ability to align auditory and textual information.
  arXiv  Detail & Related papers  (2024-05-03T14:35:58Z)
• ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversion [49.617722668505834]
  We show that our approach permits the application of speech synthesis and voice conversion to improve ASR systems using only one target-language speaker during model training. It is possible to obtain promising ASR training results with our data augmentation method using only a single real speaker in a target language.
  arXiv  Detail & Related papers  (2022-03-29T11:55:30Z)
• Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning [4.327558819000435]
  We propose a novel joint textual-phonetic pre-training approach for learning spoken language representations. Experimental results on spoken language understanding benchmarks, Fluent Speech Commands and SNIPS, show that the proposed approach significantly outperforms strong baseline models.
  arXiv  Detail & Related papers  (2021-04-21T05:19:13Z)
• Transfer Learning for Context-Aware Spoken Language Understanding [3.763434958496263]
  Spoken language understanding (SLU) is a key component of task-oriented dialogue systems. Previous work has shown that incorporating context information significantly improves SLU performance for multi-turn dialogues. We propose a Context adaptive Language Transformer (CELT) model facilitating exploiting various context information for SLU.
  arXiv  Detail & Related papers  (2020-03-03T02:56:36Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.