Bayesian Low-Rank Factorization for Robust Model Adaptation

• URL: http://arxiv.org/abs/2510.18723v1
• Date: Tue, 21 Oct 2025 15:23:30 GMT
• Title: Bayesian Low-Rank Factorization for Robust Model Adaptation
• Abstract summary: We explore factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices.<n>Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model.
• Score: 60.296574524609575
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large speech foundation models achieve strong performance across many domains, but they often require adaptation to handle local needs such as code-switching, where speakers mix languages within the same utterance. Direct fine-tuning of these models risks overfitting to the target domain and overwriting the broad capabilities of the base model. To address this challenge, we explore Bayesian factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices and thereby retain general performance while adapting to specific domains. We apply our approach to the Whisper model and evaluate on different multilingual code-switching scenarios. Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model. Compared to LoRA, our method achieves a backward gain of 54% with only a 4% drop on the new domain. These findings highlight the effectiveness of Bayesian adaptation for fine-tuning speech foundation models without sacrificing generalization.

Related papers

• Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models [6.649365031528266]
  Large multi-modal language models are increasingly deployed in high-stakes domains.<n>Traditional Bayesian methods approximate posteriors over all model weights, which becomes intractable for modern large models.<n>Recent work instead considers Bayesian low-rank adaptation to enable tractable posterior approximation.<n>Bayes Adaptation Gym (BAG) is a benchmark for the Bayesian adaptation of multi-modal language models.
  arXiv  Detail & Related papers  (2026-06-20T18:41:28Z)
• Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights [0.45880283710344066]
  Large Language Models (LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection.<n>In this study, we extend the research area of structured threat modelling by systematically evaluating domain-adapted language models.<n>We use both LLMs and Small Language Models(SLMs) that were domain adapted to telecommunications and cybersecuirty.
  arXiv  Detail & Related papers  (2026-05-11T16:31:25Z)
• Did Models Sufficient Learn? Attribution-Guided Training via Subset-Selected Counterfactual Augmentation [61.248535801314375]
  Subset-Selected Counterfactual Augmentation (SS-CA)<n>We develop Counterfactual LIMA to identify minimal spatial region sets whose removal can selectively alter model predictions.<n>Experiments show that SS-CA improves generalization on in-distribution (ID) test data and achieves superior performance on out-of-distribution (OOD) benchmarks.
  arXiv  Detail & Related papers  (2025-11-15T08:39:22Z)
• Unsupervised Parameter Efficient Source-free Post-pretraining [52.27955794126508]
  We introduce UpStep, an Unsupervised.<n>Source-free post-pretraining approach to adapt a base model from a source domain to a target domain.<n>We use various general backbone architectures, both supervised and unsupervised, trained on Imagenet as our base model.
  arXiv  Detail & Related papers  (2025-02-28T18:54:51Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR)<n>In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages.<n>We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• Mitigating Catastrophic Forgetting in Language Transfer via Model Merging [16.845734486667226]
  Branch-and-Merge (BaM) is a new adaptation method based on iteratively merging multiple models. BaM is based on the insight that this yields lower magnitude but higher quality weight changes. We demonstrate in an empirical study on Bulgarian and German that BaM can significantly reduce forgetting while matching or even improving target domain performance.
  arXiv  Detail & Related papers  (2024-07-11T17:32:40Z)
• Mitigate Domain Shift by Primary-Auxiliary Objectives Association for Generalizing Person ReID [39.98444065846305]
  ReID models struggle in learning domain-invariant representation solely through training on an instance classification objective. We introduce a method that guides model learning of the primary ReID instance classification objective by a concurrent auxiliary learning objective on weakly labeled pedestrian saliency detection. Our model can be extended with the recent test-time diagram to form the PAOA+, which performs on-the-fly optimization against the auxiliary objective.
  arXiv  Detail & Related papers  (2023-10-24T15:15:57Z)
• Robustness, Evaluation and Adaptation of Machine Learning Models in the Wild [4.304803366354879]
  We study causes of impaired robustness to domain shifts and present algorithms for training domain robust models. A key source of model brittleness is due to domain overfitting, which our new training algorithms suppress and instead encourage domain-general hypotheses.
  arXiv  Detail & Related papers  (2023-03-05T21:41:16Z)
• Variational Model Perturbation for Source-Free Domain Adaptation [64.98560348412518]
  We introduce perturbations into the model parameters by variational Bayesian inference in a probabilistic framework. We demonstrate the theoretical connection to learning Bayesian neural networks, which proves the generalizability of the perturbed model to target domains.
  arXiv  Detail & Related papers  (2022-10-19T08:41:19Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.