Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions

• URL: http://arxiv.org/abs/2602.05709v1
• Date: Thu, 05 Feb 2026 14:36:44 GMT
• Title: Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
• Abstract summary: Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices.<n>Standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns.<n>We propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation.
• Score: 22.44125988569473
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA-1519.

Related papers

• \k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating [24.538290063991997]
  Low-Rank Adaptation (LoRA) is a technique for efficient neural network fine-tuning.<n>kappa-LoRA focuses updates on the matrices with the largest condition numbers.<n>kappa-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost.
  arXiv  Detail & Related papers  (2026-07-24T17:00:40Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• TLoRA: Task-aware Low Rank Adaptation of Large Language Models [8.568676581408239]
  Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models.<n>Existing LoRA variants address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency.<n>We present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimalizes and resource allocation at the outset of training.
  arXiv  Detail & Related papers  (2026-04-20T11:43:55Z)
• Less is More: Resource-Efficient Low-Rank Adaptation [15.883867662707743]
  EffiLoRA is a lightweight and generalizable approach for language, multimodal, and diffusion models.<n>It consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation.
  arXiv  Detail & Related papers  (2025-11-30T12:52:04Z)
• HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance [27.391727025825546]
  Low-Rank Adaptation (LoRA) has emerged as a promising approach to fine-tuning large language models.<n>We propose HyperAdaLoRA, a novel framework that accelerates the convergence of AdaLoRA by leveraging a hypernetwork.<n>Our method achieves faster convergence without sacrificing performance.
  arXiv  Detail & Related papers  (2025-10-03T00:15:59Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models [0.0]
  Low-Rank Adaptation (LoRA) is a technique for reducing the number of trainable parameters by applying low-rank updates to pretrained weights.<n>We show that QR-LoRA matches or exceeds the performance of full fine-tuning, standard LoRA, and SVD-LoRA.
  arXiv  Detail & Related papers  (2025-08-29T17:47:27Z)
• Uni-LoRA: One Vector is All You Need [21.893406288629734]
  Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models.<n>Recent works such as Tied-LoRA, VeRA, and VB-LoRA push efficiency further by introducing additional constraints to reduce the trainable parameter space.<n>We show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified framework, Uni-LoRA.
  arXiv  Detail & Related papers  (2025-06-01T03:00:09Z)
• RaSA: Rank-Sharing Low-Rank Adaptation [67.40422142257091]
  Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs)<n>We introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers.<n>Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks.
  arXiv  Detail & Related papers  (2025-03-16T17:16:36Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? [121.0693322732454]
  Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.<n>Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.<n>We introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of low-rank matrices.
  arXiv  Detail & Related papers  (2024-07-25T17:57:12Z)
• SBoRA: Low-Rank Adaptation with Regional Weight Updates [19.15481369459963]
  This paper introduces Standard Basis LoRA (SBoRA), a novel parameter-efficient fine-tuning approach for Large Language Models. SBoRA reduces the number of trainable parameters by half or doubles the rank with the similar number of trainable parameters as LoRA. Our results demonstrate the superiority of SBoRA-FA over LoRA in various fine-tuning tasks, including commonsense reasoning and arithmetic reasoning.
  arXiv  Detail & Related papers  (2024-07-07T15:37:13Z)
• LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters [11.23006032094776]
  We introduce LoRA-XS, a novel fine-tuning method backed by a theoretical derivation.<n>LoRA-XS drastically reduces trainable parameters by incorporating a small, trainable weight matrix.<n>It can scale from a single parameter per module to arbitrarily large values, adapting to any storage or computational constraint.
  arXiv  Detail & Related papers  (2024-05-27T19:07:13Z)
• Asymmetry in Low-Rank Adapters of Foundation Models [47.310550805920585]
  This paper characterizes and leverages unexpected asymmetry in the importance of low-rank adapter matrices. We show that fine-tuning $B$ is inherently more effective than fine-tuning $A$, and that a random untrained $A$ should perform nearly as well as a fine-tuned one.
  arXiv  Detail & Related papers  (2024-02-26T18:59:12Z)
• LoTR: Low Tensor Rank Weight Adaptation [47.4904143988667]
  We introduce LoTR, a novel approach for parameter-efficient fine-tuning of large language models (LLMs) LoTR represents a gradient update to parameters in a form of tensor decomposition. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models.
  arXiv  Detail & Related papers  (2024-02-02T13:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.