K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs

• URL: http://arxiv.org/abs/2502.18461v2
• Date: Sun, 02 Mar 2025 12:44:06 GMT
• Title: K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs
• Authors: Ziheng Ouyang, Zhen Li, Qibin Hou,
• Abstract summary: We argue that the intrinsic properties of LoRA can effectively guide diffusion models in merging learned subject and style.<n>In each attention layer, K-LoRA compares the Top-K elements in each LoRA to be fused, determining which LoRA to select for optimal fusion.<n> Experimental results demonstrate that the proposed method effectively integrates the subject and style information learned by the original LoRAs.
• Score: 25.998945090549874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies have explored combining different LoRAs to jointly generate learned style and content. However, existing methods either fail to effectively preserve both the original subject and style simultaneously or require additional training. In this paper, we argue that the intrinsic properties of LoRA can effectively guide diffusion models in merging learned subject and style. Building on this insight, we propose K-LoRA, a simple yet effective training-free LoRA fusion approach. In each attention layer, K-LoRA compares the Top-K elements in each LoRA to be fused, determining which LoRA to select for optimal fusion. This selection mechanism ensures that the most representative features of both subject and style are retained during the fusion process, effectively balancing their contributions. Experimental results demonstrate that the proposed method effectively integrates the subject and style information learned by the original LoRAs, outperforming state-of-the-art training-based approaches in both qualitative and quantitative results.

Related papers

• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• Cached Multi-Lora Composition for Multi-Concept Image Generation [10.433033595844442]
  Low-Rank Adaptation (LoRA) has emerged as a widely adopted technique in text-to-image models.<n>Current approaches face significant challenges when composing these LoRAs for multi-concept image generation.<n>We introduce a novel, training-free framework, Cached Multi-LoRA (CMLoRA), designed to efficiently integrate multiple LoRAs.
  arXiv  Detail & Related papers  (2025-02-07T13:41:51Z)
• Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning [53.98941571078398]
  Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity.<n>Recent works adopt Mixture of Experts (MoE) by treating each LoRA module as an expert, thereby mitigating task interference through multiple specialized LoRA modules.<n>While effective, these methods often isolate knowledge within individual tasks, failing to fully exploit the shared knowledge across related tasks.<n>We propose Single-ranked Mixture of Experts LoRA (textbfSMoRA), which embeds MoE into LoRA by textittreating each rank as an
  arXiv  Detail & Related papers  (2025-01-25T06:56:39Z)
• UnZipLoRA: Separating Content and Style from a Single Image [16.61595725708187]
  UnZipLoRA is a method for decomposing an image into its constituent subject and style. UnZipLoRA disentangles these elements from a single image by training both the LoRAs simultaneously.
  arXiv  Detail & Related papers  (2024-12-05T18:59:50Z)
• CopRA: A Progressive LoRA Training Strategy [9.847045610578073]
  Low-Rank Adaptation (LoRA) is a parameter-efficient technique for fine-tuning foundation models. In this work, we propose a novel progressive training strategy for LoRA with random layer dropping. We refer to this method as Cooperative LoRA (CopRA)
  arXiv  Detail & Related papers  (2024-10-30T11:07:09Z)
• Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering [35.54018186415654]
  Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains. Existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. We propose the LoRA-LEGO framework, which conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters.
  arXiv  Detail & Related papers  (2024-09-24T15:08:41Z)
• Mixture of LoRA Experts [87.50120181861362]
  This paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach achieves superior LoRA fusion performance in comparison to direct arithmetic merging.
  arXiv  Detail & Related papers  (2024-04-21T11:59:53Z)
• Multi-LoRA Composition for Image Generation [107.83002438126832]
  We study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis.
  arXiv  Detail & Related papers  (2024-02-26T18:59:18Z)
• LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild [76.67343971195267]
  Low-Rank Adaptation (LoRA) provides an efficient solution for fine-tuning large language models (LLM) LoraRetriever is a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. Experimental results indicate that LoraRetriever consistently outperforms the baselines.
  arXiv  Detail & Related papers  (2024-02-15T15:02:46Z)
• ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs [56.85106417530364]
  Low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs. Experiments show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity.
  arXiv  Detail & Related papers  (2023-11-22T18:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.