LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation

• URL: http://arxiv.org/abs/2501.16559v2
• Date: Tue, 04 Feb 2025 18:43:24 GMT
• Title: LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation
• Authors: Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli,
• Abstract summary: A new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), enables the training-free transfer of LoRA parameters across source and target models. Our experiments demonstrate the effectiveness of LoRA-X for text-to-image generation.
• Score: 48.22550575107633
• License:
• Abstract: The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter's transferability are restricted to the target base model's weights and subspace. To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.

Related papers

• LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization [0.0]
  Low-Rank Adaptation (LoRA) and other parameter-efficient fine-tuning (PEFT) methods provide low-memory, storage-efficient solutions for personalizing text-to-image models. We show that training a hypernetwork model to generate LoRA weights can achieve competitive quality for specific domains.
  arXiv  Detail & Related papers  (2024-12-03T10:17:15Z)
• Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs [76.40876036912537]
  Large Language Models (LLMs) demonstrate strong few-shot adaptability without requiring fine-tuning. Current Visual Foundation Models (VFMs) require explicit fine-tuning with sufficient tuning data. We propose a framework, LoRA Recycle, that distills a meta-LoRA from diverse pre-tuned LoRAs with a meta-learning objective.
  arXiv  Detail & Related papers  (2024-12-03T07:25:30Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [76.11938177294178]
  We study how different fine-tuning methods change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that full fine-tuning and LoRA yield weight matrices whose singular value decompositions exhibit very different structure. We conclude by examining why intruder dimensions appear in LoRA fine-tuned models, why they are undesirable, and how their effects can be minimized.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)
• Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models [38.197552424549514]
  Low-rank adaptations (LoRAs) have revolutionized the finetuning of large foundation models. LoRAs present opportunities for applying machine learning techniques that take these low-rank weights themselves as inputs. In this paper, we investigate the potential of Learning on LoRAs (LoL), a paradigm where LoRA weights serve as input to machine learning models.
  arXiv  Detail & Related papers  (2024-10-05T15:52:47Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Are Logistic Models Really Interpretable? [10.239620808245448]
  One of the simplest AI classification models, Logistic Regression, has an unwieldy interpretation of its model weights. We show via a User Study that skilled participants are unable to reliably reproduce the action of small LR models. We argue that Linearised Additive Models (LAMs) are more interpretable than logistic models.
  arXiv  Detail & Related papers  (2024-06-19T10:36:38Z)
• $\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning [28.668326340001695]
  Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning techniques. When the base model needs to be replaced with a new one, all the associated LoRA modules need to be re-trained. This is especially problematic for commercial cloud applications where the LoRA modules and the base models are hosted by service providers.
  arXiv  Detail & Related papers  (2024-05-27T15:15:08Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• The Expressive Power of Low-Rank Adaptation [11.371811534310078]
  Low-Rank Adaptation, a parameter-efficient fine-tuning method, has emerged as a prevalent technique for fine-tuning pre-trained models. This paper takes the first step to bridge the gap by theoretically analyzing the expressive power of LoRA. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-$(fractextembedding size2)$ LoRA.
  arXiv  Detail & Related papers  (2023-10-26T16:08:33Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.