W2T: LoRA Weights Already Know What They Can Do

• URL: http://arxiv.org/abs/2603.15990v1
• Date: Mon, 16 Mar 2026 23:01:00 GMT
• Title: W2T: LoRA Weights Already Know What They Can Do
• Abstract summary: We propose a method that maps each LoRA update to a provably canonical form via QR decomposition.<n>The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding.<n>Across language and vision LoRA collections, W2T achieves strong results on classification, performance prediction, and adapter retrieval.
• Score: 27.25969992553623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.

Related papers

• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• Activated LoRA: Fine-tuned LLMs for Intrinsics [6.057520371260868]
  Low-Rank Adaptation (LoRA) has emerged as a highly efficient framework for finetuning the weights of large foundation models.<n>We propose Activated LoRA, an adapter architecture which modifies the LoRA framework to only adapt weights for the tokens in the sequence after the aLoRA is invoked.
  arXiv  Detail & Related papers  (2025-04-16T18:03:21Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks [73.09643674975591]
  Low-Rank Adaptation (LoRA) is a technique for parameter-efficient fine-tuning of Large Language Models (LLMs)<n>We study how different LoRA modules can be merged to achieve skill composition.
  arXiv  Detail & Related papers  (2024-10-16T20:33:06Z)
• Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers [0.0]
  We show how Transformers can be interpreted as dense Expectation-Maximization algorithms performed on Bayesian Nets. We propose a new model design paradigm, namely Vertical LoRA, which reduces the parameter count dramatically while preserving performance. The results show that 1) with VLoRA, the Transformer model parameter count can be reduced dramatically and 2) the performance of the original model is preserved.
  arXiv  Detail & Related papers  (2024-06-13T16:51:33Z)
• LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks [72.88244322513039]
  LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain. We propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights.
  arXiv  Detail & Related papers  (2024-02-18T04:41:25Z)
• DoRA: Weight-Decomposed Low-Rank Adaptation [57.68678247436207]
  We introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA) DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning.
  arXiv  Detail & Related papers  (2024-02-14T17:59:34Z)
• Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning [31.036465632204663]
  We introduce Chain of LoRA, an iterative optimization framework inspired by the Frank-Wolfe algorithm. We demonstrate that COLA can consistently outperform LoRA without additional computational or memory costs.
  arXiv  Detail & Related papers  (2024-01-08T14:26:49Z)
• The Expressive Power of Low-Rank Adaptation [11.371811534310078]
  Low-Rank Adaptation, a parameter-efficient fine-tuning method, has emerged as a prevalent technique for fine-tuning pre-trained models. This paper takes the first step to bridge the gap by theoretically analyzing the expressive power of LoRA. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-$(fractextembedding size2)$ LoRA.
  arXiv  Detail & Related papers  (2023-10-26T16:08:33Z)
• Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights? [81.77457373726736]
  Given a robust model trained to be resilient to one or multiple types of distribution shifts, how is that "robustness" encoded in the model weights? We propose a minimalistic model robustness "patching" framework that carries a model trained on clean data together with its pre-extracted RWSs. In this way, injecting certain robustness to the model is reduced to directly adding the corresponding RWS to its weight.
  arXiv  Detail & Related papers  (2023-02-24T06:44:19Z)
• DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation [18.922066770467914]
  Low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training. We introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together.
  arXiv  Detail & Related papers  (2022-10-14T06:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.