EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning

• URL: http://arxiv.org/abs/2608.01142v2
• Date: Tue, 04 Aug 2026 11:30:25 GMT
• Title: EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning
• Abstract summary: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning but standard LoRA produces a single deterministic model.<n>EulerLoRA generates multiple trajectories by sampling structured along the rank-one components of shared low-rank adapters.<n>EulerLoRA achieves comparable or improved performance relative to strong LoRA-Ensemble baselines.
• Score: 5.989831419348467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning, but standard LoRA produces a single deterministic model and does not directly support predictive uncertainty estimation. We introduce EulerLoRA, a stochastic extension of LoRA that generates multiple predictive trajectories by sampling structured variations along the rank-one components of shared low-rank adapters, while preserving the deterministic LoRA transformation in expectation. We evaluate EulerLoRA with vision transformers on CIFAR-10, CIFAR-100, and HAM10000, together with out-of-distribution detection on SVHN. Across these benchmarks, EulerLoRA achieves comparable or improved performance relative to strong LoRA-Ensemble baselines. Using two rank-20 adapters, EulerLoRA requires approximately 3 million trainable adapter parameters, compared with about 10 million for a rank-8, 16-adapter LoRA-Ensemble, corresponding to roughly 69% fewer trainable parameters. These results show that useful predictive diversity can be obtained from a small number of shared adapters.

Related papers

• Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence [30.96331195818476]
  Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models.<n>We introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold.<n>BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-05-29T16:08:28Z)
• HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models [11.336343540603616]
  Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning of large language models.<n>We propose Hot-Experts Layer-level Low-Rank Adaptation (HELLoRA)<n>HELLoRA attaches LoRA modules only to the most frequently activated experts at each layer.
  arXiv  Detail & Related papers  (2026-05-11T06:43:14Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• DiffLoRA: Differential Low-Rank Adapters for Large Language Models [59.58987161199141]
  We introduce DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism, with low-rank adapters on both positive and negative attention terms.<n>We evaluate DiffLoRA across a broad range of NLP tasks, including general benchmarks, many-shot in-context learning, RAG, and long-context tests.
  arXiv  Detail & Related papers  (2025-07-31T14:24:59Z)
• ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs [0.46040036610482665]
  This paper introduces Adaptive Rank Dynamic LoRA (ARD-LoRA), a novel framework that automates rank allocation through learnable scaling factors.<n>ARD-LoRA enables continuous, differentiable, per-head rank adaptation.<n> Experiments on LLAMA-3.1-70B and PaliGemma-2 demonstrate ARD-LoRA's efficacy, achieving up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters.
  arXiv  Detail & Related papers  (2025-06-23T03:45:37Z)
• DenseLoRA: Dense Low-Rank Adaptation of Large Language Models [14.133511131962786]
  Low-rank adaptation (LoRA) has been developed as an efficient approach for adapting large language models (LLMs)<n>We introduce Dense Low-Rank Adaptation (DenseLoRA), a novel approach that enhances parameter efficiency while achieving superior performance compared to LoRA.<n>We evaluate DenseLoRA on various benchmarks, showing that it achieves 83.8% accuracy with only 0.01% of trainable parameters, compared to LoRA's 80.8% accuracy with 0.70% of trainable parameters on LLaMA3-8B.
  arXiv  Detail & Related papers  (2025-05-27T08:19:07Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• Unlocking the Global Synergies in Low-Rank Adapters [20.32980343066711]
  Low-rank Adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a light-weight search algorithm that leverages zero-cost proxies to allocate the limited LoRA trainable parameters. Experiments show that HeteroLoRA enables improvements in model performance given the same parameter budge.
  arXiv  Detail & Related papers  (2024-06-21T08:10:03Z)
• LoRA: Low-Rank Adaptation of Large Language Models [71.75808607987281]
  Low-Rank Adaptation, or LoRA, freezes the pre-trained model weights and injects trainable rank decomposition into each layer of the Transformer architecture. For GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the computation hardware requirement by 3 times compared to full fine-tuning.
  arXiv  Detail & Related papers  (2021-06-17T17:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.