ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs

• URL: http://arxiv.org/abs/2311.13600v1
• Date: Wed, 22 Nov 2023 18:59:36 GMT
• Title: ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs
• Authors: Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, Varun Jampani
• Abstract summary: Low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs. Experiments show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity.
• Score: 56.85106417530364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Methods for finetuning generative models for concept-driven personalization generally achieve strong results for subject-driven or style-driven generation. Recently, low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. While recent work explores the combination of separate LoRAs to achieve joint generation of learned styles and subjects, existing techniques do not reliably address the problem; they often compromise either subject fidelity or style fidelity. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs in order to achieve generation of any user-provided subject in any user-provided style. Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize. Project page: https://ziplora.github.io

Related papers

• ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation [4.07532985236519]
  This study introduces an approach to optimize Efficient Fine Tuning (PEFT) for Pretrained Language Models (PLMs) by implementing a Shared Low Rank Adaptation (ShareLoRA) By strategically deploying ShareLoRA across different layers and adapting it for the Query, Key, and Value components of self-attention layers, we achieve a substantial reduction in the number of training parameters and memory usage. Our findings affirm that ShareLoRA effectively boosts parameter efficiency while ensuring scalable and high-quality performance across different language model architectures.
  arXiv  Detail & Related papers  (2024-06-16T02:52:28Z)
• Mixture of LoRA Experts [87.50120181861362]
  This paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach achieves superior LoRA fusion performance in comparison to direct arithmetic merging.
  arXiv  Detail & Related papers  (2024-04-21T11:59:53Z)
• CLoRA: A Contrastive Approach to Compose Multiple LoRA Models [44.037664077117945]
  Low-Rank Adaptations (LoRAs) have emerged as a powerful and popular technique in the field of image generation. CLoRA addresses the problem of seamlessly blending multiple concept LoRAs to capture a variety of concepts in one image. Our method enables the creation of composite images that truly reflect the characteristics of each LoRA.
  arXiv  Detail & Related papers  (2024-03-28T18:58:43Z)
• Implicit Style-Content Separation using B-LoRA [61.664293840163865]
  We introduce B-LoRA, a method that implicitly separate the style and content components of a single image. By analyzing the architecture of SDXL combined with LoRA, we find that jointly learning the LoRA weights of two specific blocks achieves style-content separation.
  arXiv  Detail & Related papers  (2024-03-21T17:20:21Z)
• Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation [2.2356314962198836]
  The objective of personalization and stylization in text-to-image is to instruct a pre-trained diffusion model to analyze new concepts introduced by users and incorporate them into expected styles. We propose block-wise Low-Rank Adaptation (LoRA) to perform fine-grained fine-tuning for different blocks of SD.
  arXiv  Detail & Related papers  (2024-03-12T10:38:03Z)
• Personalized LoRA for Human-Centered Text Understanding [15.704545145736676]
  We introduce personalized LoRA (PLoRA) with a plug-and-play framework for the HCTU task. PLoRA is effective, parameter-efficient and dynamically deploying in PLMs. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios.
  arXiv  Detail & Related papers  (2024-03-10T13:04:54Z)
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption [96.59370314485074]
  We propose ResLoRA, an improved framework of low-rank adaptation (LoRA) Our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-02-28T04:33:20Z)
• StyleAdapter: A Single-Pass LoRA-Free Model for Stylized Image Generation [97.24936247688824]
  This paper presents a LoRA-free method for stylized image generation that takes a text prompt and style reference images as inputs. StyleAdapter can generate high-quality images that match the content of the prompts and adopt the style of the references in a single pass.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)
• Style Interleaved Learning for Generalizable Person Re-identification [69.03539634477637]
  We propose a novel style interleaved learning (IL) framework for DG ReID training. Unlike conventional learning strategies, IL incorporates two forward propagations and one backward propagation for each iteration. We show that our model consistently outperforms state-of-the-art methods on large-scale benchmarks for DG ReID.
  arXiv  Detail & Related papers  (2022-07-07T07:41:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.