Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models

• URL: http://arxiv.org/abs/2606.22188v1
• Date: Sat, 20 Jun 2026 18:41:28 GMT
• Title: Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
• Abstract summary: Large multi-modal language models are increasingly deployed in high-stakes domains.<n>Traditional Bayesian methods approximate posteriors over all model weights, which becomes intractable for modern large models.<n>Recent work instead considers Bayesian low-rank adaptation to enable tractable posterior approximation.<n>Bayes Adaptation Gym (BAG) is a benchmark for the Bayesian adaptation of multi-modal language models.
• Score: 6.649365031528266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large multi-modal language models are increasingly deployed in high-stakes domains, making well-calibrated uncertainty essential. Traditional Bayesian methods approximate posteriors over all model weights, which becomes intractable for modern large models. For this reason, recent work instead considers Bayesian low-rank adaptation to enable tractable posterior approximation. Due to a lack of a standardized benchmark to evaluate these approaches, it remains unclear where these methods provide meaningful benefits. To fill this gap, we introduce Bayesian Adaptation Gym (BAG), a benchmark for the Bayesian adaptation of multi-modal language models. BAG provides reference implementations of classic Bayesian baselines and state-of-the-art adaptation methods, along with a multi-modal dataset and task suite designed to probe calibration, robustness under distribution shift, and decision-making under uncertainty via active learning. Using BAG, we conduct and report extensive experiments across model sizes, datasets, and tasks to highlight the successes and failures of current Bayesian adaptation approaches. To enable further research, BAG is fully open source: https://github.com/SRI-CSL/BayesAdapt.

Related papers

• Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation [69.7036779439312]
  Black box domain adaptation represents a highly practical yet extremely challenging setting.<n>We propose a dual teacher distillation with subnetwork rectification (DDSR) model that exploits specific knowledge embedded in black box source models.<n>Our approach demonstrates consistent improvements over state of the art methods, including those using source data or models.
  arXiv  Detail & Related papers  (2026-03-24T07:54:19Z)
• Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendation [60.33386541343322]
  We propose a Multimodal Large Language Models framework that integrates Hardness-aware and Noise-regularized preference optimization for Recommendation (HaNoRec)<n>Specifically, HaNoRec dynamically adjusts optimization weights based on both the estimated hardness of each training sample and the policy model's real-time responsiveness.
  arXiv  Detail & Related papers  (2025-11-24T04:10:46Z)
• Bayesian Low-Rank Factorization for Robust Model Adaptation [60.296574524609575]
  We explore factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices.<n>Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model.
  arXiv  Detail & Related papers  (2025-10-21T15:23:30Z)
• Training-Free Bayesianization for Low-Rank Adapters of Large Language Models [22.80864697991734]
  Training-Free Bayesianization (TFB) is a framework that transforms trained low-rank adapters into Bayesian ones without additional training.<n>TFB achieves superior uncertainty estimation and generalization compared to existing methods.
  arXiv  Detail & Related papers  (2024-12-07T18:49:27Z)
• Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
  Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. We propose a test-time Forward-Optimization Adaptation (FOA) method. FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
  arXiv  Detail & Related papers  (2024-04-02T05:34:33Z)
• Bayesian Exploration Networks [28.885750299203433]
  We introduce a novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies. As a first step towards model-free Bayes optimality, we introduce the Bayesian exploration network (BEN) which uses normalising flows to model both the aleatoric uncertainty (via density estimation) and epistemic uncertainty (via variational inference) in the Bellman operator. In the limit of complete optimisation, BEN learns true Bayes-optimal policies, but like in variational expectation-maximisation, partial optimisation renders our approach tractable.
  arXiv  Detail & Related papers  (2023-08-24T19:35:58Z)
• Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift [19.945634052291542]
  We evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks. We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures. We provide the first systematic evaluation of BDL for fine-tuning large pre-trained models.
  arXiv  Detail & Related papers  (2023-06-21T14:36:03Z)
• On Pitfalls of Test-Time Adaptation [82.8392232222119]
  Test-Time Adaptation (TTA) has emerged as a promising approach for tackling the robustness challenge under distribution shifts. We present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols.
  arXiv  Detail & Related papers  (2023-06-06T09:35:29Z)
• Adaptive variational Bayes: Optimality, computation and applications [2.4022340214033915]
  We propose a novel adaptive variational Bayes framework, which can operate on a collection of models. We show that the adaptive variational Bayes attains optimal contraction rates adaptively under very general conditions.
  arXiv  Detail & Related papers  (2021-09-07T17:18:28Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.