Variational Low-Rank Adaptation Using IVON

• URL: http://arxiv.org/abs/2411.04421v2
• Date: Sat, 09 Nov 2024 12:30:06 GMT
• Title: Variational Low-Rank Adaptation Using IVON
• Abstract summary: We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models.
• Score: 50.0349688602232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at https://github.com/team-approx-bayes/ivon-lora.

Related papers

• M+Adam: Low-Precision Training via Additive-Multiplicative Optimization [69.5989114185868]
  Training with quantized weights can reduce costs but often results in degraded accuracy.<n>We propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes.<n>Across LLaMA-style pretraining with 60M-1B models, 1x-8x Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.
  arXiv  Detail & Related papers  (2026-07-12T07:15:47Z)
• Improving LoRA with Variational Learning [21.194015348736066]
  We show that variational learning with IVON can effectively improve LoRA finetuning.<n>We present extensive results on billion-scale LLMs (Llama and Qwen series) going way beyond the scale of existing applications of IVON.
  arXiv  Detail & Related papers  (2025-06-17T07:49:43Z)
• Variational Visual Question Answering [28.075269845580056]
  We propose a Variational VQA approach to enhance reliability of multimodal models.<n>We employ a proposed variational algorithm called IVON, which yields a posterior distribution over model parameters.<n>Our approach improves calibration and abstentions without sacrificing accuracy of AdamW.
  arXiv  Detail & Related papers  (2025-05-14T17:40:22Z)
• A hybrid framework for effective and efficient machine unlearning [12.499101994047862]
  Machine unlearning (MU) is proposed to remove the imprints of revoked samples from the already trained model parameters. We present a novel hybrid strategy on top of them to achieve an overall success.
  arXiv  Detail & Related papers  (2024-12-19T03:59:26Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR) In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.47014540413659]
  We propose a unified training framework for deep neural networks. We introduce three instances of MARS that leverage preconditioned gradient optimization. Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models [3.7049613588433497]
  Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. We extend the LoRA to multiple scales, dubbed as LoRA$2$.
  arXiv  Detail & Related papers  (2024-08-13T12:31:30Z)
• COBias and Debias: Minimizing Language Model Pairwise Accuracy Bias via Nonlinear Integer Programming [12.287692969438169]
  We tackle language models' imbalance in per-class prediction accuracy by reconceptualizing it as the Contextuality Bias (COBias) We are the first to engage nonlinear integer programming (NIP) to debias it. DNIP simultaneously achieves significant COBias reduction and accuracy improvement over the conventional ICL approach.
  arXiv  Detail & Related papers  (2024-05-13T10:30:33Z)
• Variational Learning is Effective for Large Deep Networks [76.94351631300788]
  We show that an Improved Variational Online Newton consistently matches or outperforms Adam for training large networks. IVON's computational costs are nearly identical to Adam but its predictive uncertainty is better. We find overwhelming evidence that variational learning is effective.
  arXiv  Detail & Related papers  (2024-02-27T16:11:05Z)
• Neural Priming for Sample-Efficient Adaptation [92.14357804106787]
  We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks. Neural Priming can be performed at test time, even for pretraining as large as LAION-2B.
  arXiv  Detail & Related papers  (2023-06-16T21:53:16Z)
• A Close Look into the Calibration of Pre-trained Language Models [56.998539510508515]
  Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. We study the dynamic change in PLMs' calibration performance in training. We extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations.
  arXiv  Detail & Related papers  (2022-10-31T21:31:07Z)
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level [38.64433236359172]
  We find that BERT-Large is worse than BERT-Mini on at least 1-4% of instances across MNLI, SST-2, and QQP. Finetuning noise increases with model size and that instance-level accuracy has momentum. Our findings suggest that instance-level predictions provide a rich source of information.
  arXiv  Detail & Related papers  (2021-05-13T01:10:51Z)
• BayCANN: Streamlining Bayesian Calibration with Artificial Neural Network Metamodeling [77.34726150561087]
  We propose to use artificial neural networks (ANN) as one solution to limitations of Bayesian calibration. We demonstrate BayCANN by calibrating a natural history model of colorectal cancer to adenoma prevalence and cancer incidence data. BayCANN was generally more accurate than IMIS in recovering the "true" values.
  arXiv  Detail & Related papers  (2020-10-26T09:47:39Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.