AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing

• URL: http://arxiv.org/abs/2512.22455v2
• Date: Mon, 05 Jan 2026 04:00:54 GMT
• Title: AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
• Abstract summary: AFA-LoRA is a novel training strategy that brings non-linear expressivity to LoRA while maintaining its seamless mergeability.<n>We implement our method on supervised fine-tuning, reinforcement learning, and speculative decoding.
• Score: 22.025349145835182
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method. However, its linear adaptation process limits its expressive power. This means there is a gap between the expressive power of linear training and non-linear training. To bridge this gap, we propose AFA-LoRA, a novel training strategy that brings non-linear expressivity to LoRA while maintaining its seamless mergeability. Our key innovation is an annealed activation function that transitions from a non-linear to a linear transformation during training, allowing the adapter to initially adopt stronger representational capabilities before converging to a mergeable linear form. We implement our method on supervised fine-tuning, reinforcement learning, and speculative decoding. The results show that AFA-LoRA reduces the performance gap between LoRA and full-parameter training. This work enables a more powerful and practical paradigm of parameter-efficient adaptation.

Related papers

• Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints [22.829834841445603]
  In gated FFNs, a low-rank residual can change not only projected features but also the nonlinear selection weights that determine which channels contribute to the output.<n>We formalize this effect as selection misalignment and connect it to the local effective homogeneity of self-gated activations.<n>We propose NA-LoRA, a training-only method with two lightweight mechanisms.
  arXiv  Detail & Related papers  (2026-06-30T14:21:51Z)
• JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models [78.56572636138429]
  We propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating.<n>We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
  arXiv  Detail & Related papers  (2026-04-17T15:38:37Z)
• Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations [25.21314821550254]
  Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models.<n>We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates.<n> Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA.
  arXiv  Detail & Related papers  (2025-09-26T04:54:02Z)
• Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning [82.16625951603315]
  NoRA replaces fixed activations with learnable rational functions and applies structured low-rank updates to numerator and denominator coefficients.<n>On vision transformers trained on CIFAR-10 and CIFAR-100, NoRA matches or exceeds full fine-tuning while updating only 0.4% of parameters.<n>NoRA constrains adaptation to a low-dimensional functional subspace, implicitly regularizing update magnitude and direction.
  arXiv  Detail & Related papers  (2025-09-16T16:47:03Z)
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping [28.37735374308455]
  AuroRA: (I) not only matches or surpasses full fine-tuning performance with only 6.18% 25% of LoRA's parameters but also (II) outperforms state-of-the-art PEFT methods by up to 10.88% in both NLP and CV tasks.
  arXiv  Detail & Related papers  (2025-05-24T15:16:27Z)
• DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation [44.99833362998488]
  Decoupled Low-rank Adaptation (DeLoRA) is a novel finetuning method that normalizes and scales learnable low-rank matrices.<n>We show that DeLoRA matches or surpasses performance of competing PEFT methods, while exhibiting stronger robustness.
  arXiv  Detail & Related papers  (2025-03-23T22:00:56Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs [75.11449420928139]
  Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a promising solution, but there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. We propose eXtreme Gradient Boosting LoRA, a novel framework that bridges this gap by leveraging the power of ensemble learning.
  arXiv  Detail & Related papers  (2024-10-25T17:07:13Z)
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption [96.59370314485074]
  We propose ResLoRA, an improved framework of low-rank adaptation (LoRA) Our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-02-28T04:33:20Z)
• DoRA: Weight-Decomposed Low-Rank Adaptation [57.68678247436207]
  We introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA) DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning.
  arXiv  Detail & Related papers  (2024-02-14T17:59:34Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.