Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR

• URL: http://arxiv.org/abs/2607.11163v1
• Date: Mon, 13 Jul 2026 06:57:02 GMT
• Title: Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR
• Abstract summary: Fine-tuning on low-resource languages often causes catastrophic forgetting.<n>We propose Unified Gradient Projection, which constrains parameter updates using reference gradients from language-balanced replay.<n>On Whisper-large-v3, it achieves near-zero average forgetting.
• Score: 6.818862582399845
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.

Related papers

• Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages [3.6077805387487545]
  Onramp-Sequence Cross-Distillation (OSCD) projects high-resource reasoning trajectories into low-resource vocabulary subspaces during generative training rollouts.<n> OSCD yields up to 3.2 times overall improvements in native Southeast Asian languages for mathematical reasoning.
  arXiv  Detail & Related papers  (2026-08-01T08:44:32Z)
• GigaAM Multilingual: Foundation Model for Underrepresented Languages [63.406165255277195]
  We present GigaAM Multilingual, a Conformer encoder pre-trained on 2M hours of audio using a HuBERT-style objective.<n>We introduce a cluster-level data balancing strategy during pre-training and a domain-aware sampling method during fine-tuning to mitigate head-language dominance.
  arXiv  Detail & Related papers  (2026-07-11T15:48:03Z)
• Multilingual Fine-Tuning via Localized Gradient Conflict Resolution [20.41455787951831]
  We introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally.<n> Empirically, Bucket-Level MOOs mitigate interference by driving Large Language Models to construct distinct language-specific dimensions.<n>Our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.
  arXiv  Detail & Related papers  (2026-06-04T02:36:30Z)
• Crosslingual On-Policy Self-Distillation for Multilingual Reasoning [48.68444770923683]
  Crosslingual On-Policy Self-Distillation (COPSD) transfers a model's own high-resource reasoning behavior to low-resource languages.<n>Experiments on 17 low-resource African languages show that COPSD consistently improves low-resource mathematical reasoning across model sizes.
  arXiv  Detail & Related papers  (2026-05-10T14:06:09Z)
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training [7.141617235075284]
  Post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages.<n>We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain.<n>We find that increasing language coverage during post-training is largely beneficial across tasks and model scales.
  arXiv  Detail & Related papers  (2026-04-14T20:26:34Z)
• Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning [58.355275813623685]
  This work looks at a critical gap in the calibration of large language models (LLMs) within multilingual settings.<n>Even in low-resource languages, model confidence can increase significantly after instruction-tuning on high-resource language SFT datasets.<n>However, improvements in accuracy are marginal or non-existent, highlighting a critical shortcoming of standard SFT for multilingual languages.
  arXiv  Detail & Related papers  (2026-01-04T04:29:12Z)
• CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment [40.44120745699753]
  We propose a scalable and effective filtering method that selects high-quality preference samples with minimal gradient conflicts across languages.<n>Our method leverages gradient surgery to retain samples aligned with an aggregated multilingual update direction.<n>We integrate CONGRAD into self-rewarding framework and evaluate on LLaMA3-8B and Gemma2-2B across 10 languages.
  arXiv  Detail & Related papers  (2025-03-31T06:52:56Z)
• Weighted Cross-entropy for Low-Resource Languages in Multilingual Speech Recognition [2.7247388777405597]
  We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language.
  arXiv  Detail & Related papers  (2024-09-25T14:09:09Z)
• SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition [55.2480439325792]
  Speech Meta In-Context LEarning (SMILE) is an innovative framework that combines meta-learning with speech in-context learning (SICL)<n>We show that SMILE consistently outperforms baseline methods in training-free few-shot multilingual ASR tasks.
  arXiv  Detail & Related papers  (2024-09-16T16:04:16Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference [54.678516076366506]
  Natural Language Inference (NLI) is a growingly essential task in natural language understanding. Here we propose a multi-level supervised contrastive learning framework named MultiSCL for low-resource natural language inference.
  arXiv  Detail & Related papers  (2022-05-31T05:54:18Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• Demystify Optimization Challenges in Multilingual Transformers [21.245418118851884]
  We study optimization challenges from loss landscape and parameter plasticity perspectives. We find that imbalanced training data poses task interference between high and low resource languages. We propose Curvature Aware Task Scaling (CATS) which improves both optimization and generalization especially for low resource.
  arXiv  Detail & Related papers  (2021-04-15T17:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.