Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

• URL: http://arxiv.org/abs/2603.24242v2
• Date: Thu, 26 Mar 2026 11:39:50 GMT
• Title: Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition
• Abstract summary: Federated Learning of Large Language Models (LLMs) in multilingual environments presents significant challenges.<n>We extend the FederatedScope-LLM framework to support multilingual instruction-tuning experiments with LLMs.<n>We introduce a novel client-specific early stopping mechanism, Local Dynamic Early Stopping (LDES-FL), which allows clients to pause and resume local training.
• Score: 2.897376329231807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Learning (FL) of Large Language Models (LLMs) in multilingual environments presents significant challenges stemming from heterogeneous language distributions across clients and disparities in language resource availability. To address these challenges, we extended the FederatedScope-LLM framework to support multilingual instruction-tuning experiments with LLMs. We also introduced a novel client-specific early stopping mechanism, Local Dynamic Early Stopping (LDES-FL), which allows clients to pause and resume local training based on client-side validation performance, enhancing training efficiency and sustainability. Through a series of experiments, we studied how client language composition - from fully monolingual to increasingly multilingual clients - affects multilingual quality, fairness and training cost. Monolingual local fine-tuning remains the most effective for single-language specialization, whereas federated training is better suited to learning a single balanced multilingual model. In FL, increasing within-client multilinguality leads to stronger and fairer global models, narrows the gap to centralized multilingual fine-tuning, and yields the largest gains for lower-resource languages, albeit at the cost of more optimization steps. Overall, our results identify client language composition as a key design variable in multilingual FL, shaping performance, fairness and efficiency.

Related papers

• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation [50.731838216268706]
  We study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus.<n>We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers.<n>We propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers.
  arXiv  Detail & Related papers  (2026-05-28T10:12:41Z)
• MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting [69.6938830307759]
  MiLorE-SSL is a lightweight framework that combines LoRA modules with a soft mixture-of-experts mechanism for efficient continual multilingual training.<n>LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference.<n>Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters.
  arXiv  Detail & Related papers  (2026-01-28T06:48:52Z)
• Aligning LLMs for Multilingual Consistency in Enterprise Applications [39.24743635297897]
  Large language models (LLMs) remain unreliable for global enterprise applications.<n>We propose a practical, batch-wise alignment strategy for fine-tuning LLMs.<n>This approach improves non-English accuracy by up to 23.9% without compromising English performance, model reasoning, or retrieval quality.
  arXiv  Detail & Related papers  (2025-09-28T05:51:22Z)
• Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining [16.590296049892576]
  This paper introduces Climb, a novel framework designed to systematically optimize multilingual data allocation.<n>At its core, Climb introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language's effective allocation by capturing inter-language dependencies.<n>Extensive experiments confirm that Climb can accurately measure cross-lingual interactions across various multilingual settings.
  arXiv  Detail & Related papers  (2025-09-19T03:34:34Z)
• Bridging Language Gaps: Enhancing Few-Shot Language Adaptation [32.157041759856]
  The disparity in language resources poses a challenge in multilingual NLP.<n>High-resource languages benefit from extensive data, while low-resource languages lack sufficient data for effective training.<n>Our Contrastive Language Alignment with Prompting (CoLAP) method addresses this gap by integrating contrastive learning with cross-lingual representations.
  arXiv  Detail & Related papers  (2025-08-26T22:49:17Z)
• Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages [27.63253872229416]
  We propose a Federated Prompt Tuning Paradigm for multilingual scenarios.<n>Our approach achieves 6.9% higher accuracy with improved data efficiency.<n>These findings underscore the potential of our approach to promote social equality and champion linguistic diversity.
  arXiv  Detail & Related papers  (2025-07-02T05:23:20Z)
• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision.<n>LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks.<n>We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation [25.850573463743352]
  Large-scale multilingual Pretrained Language Models (mPLMs) yield impressive performance on cross-language tasks. Yet significant performance disparities exist across different languages within the same mPLM. We introduce ALSACE to leverage the learned knowledge from the well-performing languages to guide under-performing ones within the same mPLM.
  arXiv  Detail & Related papers  (2024-04-12T14:19:16Z)
• Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages [60.162717568496355]
  Large language models (LLMs) have been pre-trained on multilingual corpora. Their performance still lags behind in most languages compared to a few resource-rich languages.
  arXiv  Detail & Related papers  (2024-02-19T15:07:32Z)
• Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca [32.66997811671922]
  This study employs the Alpaca dataset and machine translations of it to form multilingual data. Under a controlled budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Our findings serve as a guide for expanding language support through instruction tuning.
  arXiv  Detail & Related papers  (2023-09-16T11:22:46Z)
• Generalizing Multimodal Pre-training into Multilingual via Language Acquisition [54.69707237195554]
  English-based Vision-Language Pre-training has achieved great success in various downstream tasks. Some efforts have been taken to generalize this success to non-English languages through Multilingual Vision-Language Pre-training. We propose a textbfMultitextbfLingual textbfAcquisition (MLA) framework that can easily generalize a monolingual Vision-Language Pre-training model into multilingual.
  arXiv  Detail & Related papers  (2022-05-29T08:53:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.