AC-LoRA: Auto Component LoRA for Personalized Artistic Style Image Generation

• URL: http://arxiv.org/abs/2504.02231v1
• Date: Thu, 03 Apr 2025 02:56:01 GMT
• Title: AC-LoRA: Auto Component LoRA for Personalized Artistic Style Image Generation
• Authors: Zhipu Cui, Andong Tian, Zhi Ying, Jialiang Lu,
• Abstract summary: AC-LoRA is able to automatically separate the signal component and noise component of the LoRA matrices for fast and efficient personalized artistic style image generation.<n>Results were validated using FID, CLIP, DINO, and ImageReward, achieving an average of 9% improvement.
• Score: 2.2820583483778045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized image generation allows users to preserve styles or subjects of a provided small set of images for further image generation. With the advancement in large text-to-image models, many techniques have been developed to efficiently fine-tune those models for personalization, such as Low Rank Adaptation (LoRA). However, LoRA-based methods often face the challenge of adjusting the rank parameter to achieve satisfactory results. To address this challenge, AutoComponent-LoRA (AC-LoRA) is proposed, which is able to automatically separate the signal component and noise component of the LoRA matrices for fast and efficient personalized artistic style image generation. This method is based on Singular Value Decomposition (SVD) and dynamic heuristics to update the hyperparameters during training. Superior performance over existing methods in overcoming model underfitting or overfitting problems is demonstrated. The results were validated using FID, CLIP, DINO, and ImageReward, achieving an average of 9% improvement.

Related papers

• HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis [11.828681423119313]
  We introduce a parameter-efficient adaptive generation method, namely HyperLoRA, that uses an adaptive plug-in network to generate LoRA weights.<n>We achieve zero-shot personalized portrait generation with high photorealism, fidelity, and editability.
  arXiv  Detail & Related papers  (2025-03-21T08:44:27Z)
• A LoRA is Worth a Thousand Pictures [28.928964530616593]
  Low Rank Adaptation (LoRA) can replicate an artist's style or subject using minimal data and computation. We show that LoRA weights alone can serve as an effective descriptor of style, without the need for additional image generation or knowledge of the original training set. We conclude with a discussion on potential future applications, such as zero-shot LoRA fine-tuning and model attribution.
  arXiv  Detail & Related papers  (2024-12-16T18:18:17Z)
• LoRACLR: Contrastive Adaptation for Customization of Diffusion Models [62.70911549650579]
  LoRACLR is a novel approach for multi-concept image generation that merges multiple LoRA models, each fine-tuned for a distinct concept, into a single, unified model.<n>LoRACLR uses a contrastive objective to align and merge the weight spaces of these models, ensuring compatibility while minimizing interference.<n>Our results highlight the effectiveness of LoRACLR in accurately merging multiple concepts, advancing the capabilities of personalized image generation.
  arXiv  Detail & Related papers  (2024-12-12T18:59:55Z)
• LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation [28.098287135605364]
  We introduce LoRA.rar, a method that improves image quality and achieves a remarkable speedup of over $4000times$ in the merging process.<n>LoRA.rar pre-trains a hypernetwork on a diverse set of content-style LoRA pairs, learning an efficient merging strategy that generalizes to new, unseen content-style pairs.<n>Our method significantly outperforms the current state of the art in both content and style fidelity, as validated by MLLM assessments and human evaluations.
  arXiv  Detail & Related papers  (2024-12-06T16:04:56Z)
• LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair [116.48684498656871]
  We propose the LoRA of Change (LoC) framework for image editing with visual instructions, i.e., before-after image pairs.<n>We learn an instruction-specific LoRA to encode the "change" in a before-after image pair, enhancing the interpretability and reusability of our model.<n>Our model produces high-quality images that align with user intent and support a broad spectrum of real-world visual instructions.
  arXiv  Detail & Related papers  (2024-11-28T13:55:06Z)
• UIR-LoRA: Achieving Universal Image Restoration through Multiple Low-Rank Adaptation [50.27688690379488]
  Existing unified methods treat multi-degradation image restoration as a multi-task learning problem. We propose a universal image restoration framework based on multiple low-rank adapters (LoRA) from multi-domain transfer learning. Our framework leverages the pre-trained generative model as the shared component for multi-degradation restoration and transfers it to specific degradation image restoration tasks.
  arXiv  Detail & Related papers  (2024-09-30T11:16:56Z)
• Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape [52.98187034726091]
  Low-Rank Adaptation (LoRA) is an efficient way to fine-tune models by optimizing only a low-rank matrix. A solution that appears flat in the LoRA space may exist sharp directions in the full parameter space, potentially harming generalization performance. We propose Flat-LoRA, an efficient approach that seeks a low-rank adaptation located in a flat region of the full parameter space.
  arXiv  Detail & Related papers  (2024-09-22T11:24:10Z)
• DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion [43.55179971287028]
  We propose DiffLoRA, an efficient method that leverages the diffusion model as a hypernetwork to predict personalized Low-Rank Adaptation weights. By incorporating these LoRA weights into the off-the-shelf text-to-image model, DiffLoRA enables zero-shot personalization during inference. We introduce a novel identity-oriented LoRA weights construction pipeline to facilitate the training process of DiffLoRA.
  arXiv  Detail & Related papers  (2024-08-13T09:00:35Z)
• TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation [5.195293792493412]
  We propose an innovative method that integrates Singular Value Decomposition into the Low-Rank Adaptation (LoRA) parameter update strategy. By incorporating SVD within the LoRA framework, our method not only effectively reduces the risk of overfitting but also enhances the stability of model outputs.
  arXiv  Detail & Related papers  (2024-05-18T09:29:00Z)
• Active Generation for Image Classification [45.93535669217115]
  We propose to address the efficiency of image generation by focusing on the specific needs and characteristics of the model. With a central tenet of active learning, our method, named ActGen, takes a training-aware approach to image generation.
  arXiv  Detail & Related papers  (2024-03-11T08:45:31Z)
• Multi-LoRA Composition for Image Generation [107.83002438126832]
  We study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis.
  arXiv  Detail & Related papers  (2024-02-26T18:59:18Z)
• DGNet: Dynamic Gradient-Guided Network for Water-Related Optics Image Enhancement [77.0360085530701]
  Underwater image enhancement (UIE) is a challenging task due to the complex degradation caused by underwater environments. Previous methods often idealize the degradation process, and neglect the impact of medium noise and object motion on the distribution of image features. Our approach utilizes predicted images to dynamically update pseudo-labels, adding a dynamic gradient to optimize the network's gradient space.
  arXiv  Detail & Related papers  (2023-12-12T06:07:21Z)
• Characteristic Regularisation for Super-Resolving Face Images [81.84939112201377]
  Existing facial image super-resolution (SR) methods focus mostly on improving artificially down-sampled low-resolution (LR) imagery. Previous unsupervised domain adaptation (UDA) methods address this issue by training a model using unpaired genuine LR and HR data. This renders the model overstretched with two tasks: consistifying the visual characteristics and enhancing the image resolution. We formulate a method that joins the advantages of conventional SR and UDA models.
  arXiv  Detail & Related papers  (2019-12-30T16:27:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.