Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms

• URL: http://arxiv.org/abs/2605.08423v1
• Date: Fri, 08 May 2026 19:32:43 GMT
• Title: Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms
• Abstract summary: We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks.<n>Our approach replaces a purely layer-local adapter with a shared queryable memory of low-rank update atoms.
• Score: 41.71279032037811
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a data-adaptive method for parameter-efficient fine-tuning of large neural networks. Standard low-rank adaptation methods improve efficiency by restricting each layer update to a fixed low-rank form, but this static parameterization can be too rigid when the appropriate correction depends on the input and on the evolving depth-wise computation of the network. Our approach replaces a purely layer-local adapter with a shared queryable memory of low-rank update atoms. For each block of layers, the model forms a query from the current low-rank state and a running summary of previous blocks, uses this query to retrieve a content-dependent combination of shared update components via attention, and applies the resulting routed operator within the low-rank bottleneck. In this way, the method retains the efficiency and scalability of low-rank adaptation while allowing the effective update to vary across inputs and to share reusable structure across layers. The resulting architecture provides a principled middle ground between static LoRA-style updates and fully generated parameter updates: it remains compact and parameter-efficient while supporting dynamic, context-sensitive adaptation. Further, we incorporate instruction-regularization by augmenting routing logits with a language-induced prior over update atoms, thereby biasing the selection of low-rank transformations toward semantically relevant directions without generating unconstrained parameter updates. Experiments on noisy non-linear regression tasks and LLM fine-tuning suggest that this queryable update-memory formulation can improve final test performance and training stability compared to standard low-rank adaptation, while using a comparable number of trainable parameters.

Related papers

• Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models [8.687019807776881]
  Training large neural networks with data-parallel gradient descent allocates N GPU replicas to compute effectively identical updates.<n>We propose a method that repurposes these replicas for simultaneous learning rate configurations overhead.<n>We further propose an automatic learning rate (autoLR) controller that treats the relative training loss across replicas as a performance signal.
  arXiv  Detail & Related papers  (2026-04-27T17:17:28Z)
• Merging Beyond: Streaming LLM Updates via Activation-Guided Rotations [55.047454145941366]
  Streaming Merging is an innovative model updating paradigm that conceptualizes merging as an iterative optimization process.<n> ARM is a strategy designed to approximate gradient descent dynamics.<n> ARM requires only early SFT checkpoints and, through iterative merging, surpasses the fully converged SFT model.
  arXiv  Detail & Related papers  (2026-02-03T08:15:57Z)
• Efficient Continual Learning in Neural Machine Translation: A Low-Rank Adaptation Approach [0.4870012761464388]
  Continual learning in Neural Machine Translation (NMT) faces the dual challenges of catastrophic forgetting and the high computational cost of retraining.<n>This study establishes Low-Rank Adaptation (LoRA) as a parameter-efficient framework to address these challenges.
  arXiv  Detail & Related papers  (2025-12-10T18:37:57Z)
• TsqLoRA: Towards Sensitivity and Quality Low-Rank Adaptation for Efficient Fine-Tuning [8.738094135297786]
  Fine-tuning large pre-trained models for downstream tasks is computationally expensive and memory-intensive.<n>We propose TsqLoRA, a novel method that integrates data-quality-driven selection with sensitivity-aware low-rank adaptation.<n>The experimental results demonstrate that TsqLoRA improves fine-tuning efficiency while maintaining or even improving performance on a variety of NLP tasks.
  arXiv  Detail & Related papers  (2025-09-23T03:10:41Z)
• Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes [3.246129789918632]
  The training of deep neural networks is inherently a non- optimization problem.<n>Standard approaches such as gradient descent (SGD) require simultaneous updates to parameters.<n>We propose a novel method Alternating Train Miniization with tailored step sizes (SAMT)<n>SAMT achieves better performance with fewer parameter updates compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-06T08:23:38Z)
• PVBF: A Framework for Mitigating Parameter Variation Imbalance in Online Continual Learning [19.18078967631654]
  Online continual learning (OCL) enables AI systems to adaptively learn from non-stationary data streams.<n>This paper identifies parameter variation imbalance as a critical factor contributing to prediction bias in ER-based OCL.
  arXiv  Detail & Related papers  (2025-02-25T02:56:42Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• LoRTA: Low Rank Tensor Adaptation of Large Language Models [70.32218116940393]
  Low Rank Adaptation (LoRA) is a popular Efficient Fine Tuning (PEFT) method.<n>We propose a higher-order Candecomp/Parafac (CP) decomposition, enabling a more compact and flexible representation.<n>Our method can achieve a reduction in the number of parameters while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-10-05T06:59:50Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.