SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation

• URL: http://arxiv.org/abs/2605.22743v1
• Date: Thu, 21 May 2026 17:13:49 GMT
• Title: SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation
• Abstract summary: We propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework.<n>We prove that learning the LoRA basis from data minimizes residual interference energy more effectively than frozen-basis methods.<n> Experiments on multi-concept image generation demonstrate that SeqLoRA improves identity preservation and scalability across up to 101 concepts.
• Score: 33.32101400652499
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models, but composing multiple custom concepts remains challenging due to representation interference. Existing modular methods either rely on expensive post-hoc fusion or freeze adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization. Theoretically, we establish strong convergence guarantees for our algorithm and model the residual layer activations as a matrix sub-Gaussian process to derive high-probability bounds on catastrophic forgetting. We further prove that learning the LoRA basis from data minimizes residual interference energy more effectively than frozen-basis methods. Experiments on multi-concept image generation demonstrate that SeqLoRA improves identity preservation and scalability across up to 101 concepts, while avoiding costly fusion and reducing attribute interference in composed generations.

Related papers

• Rarity-Aware Discrete Diffusion with Spatially Consistent Decoding for Photo-Realistic Image Super-Resolution [80.28905986756685]
  DiMOO-SR is a rarity-aware multimodal discrete diffusion framework for photo-realistic image Super-Resolution.<n>Experiments on widely used real-world SR benchmarks demonstrate that DiMOO-SR achieves competitive perceptual quality with only a few parallel decoding steps.
  arXiv  Detail & Related papers  (2026-07-20T07:01:01Z)
• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• FlowErase-RL: Rethinking Concept Erasure as Reward Optimization in Flow Matching Models [47.80885954469943]
  Flow matching models have significantly improved text-to-image generation quality, but also introduce growing safety risks.<n>We propose emphFlowErase-RL, the first GRPO-based framework for concept erasure in flow matching models.<n>We show that our method achieves state-of-the-art erasure performance while maintaining strong image quality and semantic alignment.
  arXiv  Detail & Related papers  (2026-05-19T12:10:09Z)
• UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning [3.8373805990749266]
  UnHype is a framework that incorporates hypernetworks into single- and multi-concept Low-Rank Adaptation (LoRA) training.<n>During inference, the hypernetwork dynamically generates adaptive LoRA weights based on the CLIP embedding.<n>We evaluate UnHype across several challenging tasks, including object erasure, celebrity erasure, and explicit content removal.
  arXiv  Detail & Related papers  (2026-02-03T11:37:08Z)
• MambaX: Image Super-Resolution with State Predictive Control [48.76194230142064]
  Mamba has emerged as a promising approach that can represent the entire reconstruction process as a state sequence with multiple nodes, allowing for intermediate intervention.<n>We created a nonlinear state predictive control model textbfMambaX that maps consecutive spectral bands into a latent state space and generalizes the SR task by dynamically learning the nonlinear state parameters of control equations.<n>Our evaluation demonstrates the superior performance of the dynamic spectrum-state representation model in both single-image SR and multimodal fusion-based SR tasks.
  arXiv  Detail & Related papers  (2025-11-22T11:44:09Z)
• HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking [80.07224739976911]
  Event cameras offer exceptional temporal resolution and a range (modal)<n> RGB cameras excel at capturing rich texture with high resolution, whereas event cameras offer exceptional temporal resolution and a range (modal)
  arXiv  Detail & Related papers  (2025-10-22T13:15:13Z)
• AutoLoRA: Automatic LoRA Retrieval and Fine-Grained Gated Fusion for Text-to-Image Generation [32.46570968627392]
  Low-rank adaptation (LoRA) have demonstrated efficacy in enabling model customization with minimal parameter overhead.<n>We introduce a novel framework that enables semantic-driven LoRA retrieval and dynamic aggregation.<n>Our approach achieves significant improvement in image generation perfermance.
  arXiv  Detail & Related papers  (2025-08-04T06:36:00Z)
• FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities [76.46448367752944]
  multimodal large language models (MLLMs) unify visual understanding and image generation within a single framework.<n>Most existing MLLMs rely on autore (AR) architectures, which impose inherent limitations on future development.<n>We introduce FUDOKI, a unified multimodal model purely based on discrete flow matching.
  arXiv  Detail & Related papers  (2025-05-26T15:46:53Z)
• LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models [33.379758040084894]
  Multi-concept customization emerges as the challenging task within this domain. Existing approaches often rely on training a fusion matrix of multiple Low-Rank Adaptations (LoRAs) to merge various concepts into a single image. LoRA-Composer is a training-free framework designed for seamlessly integrating multiple LoRAs.
  arXiv  Detail & Related papers  (2024-03-18T09:58:52Z)
• Convex Latent-Optimized Adversarial Regularizers for Imaging Inverse Problems [8.33626757808923]
  We introduce Convex Latent-d Adrial Regularizers (CLEAR), a novel and interpretable data-driven paradigm. CLEAR represents a fusion of deep learning (DL) and variational regularization. Our method consistently outperforms conventional data-driven techniques and traditional regularization approaches.
  arXiv  Detail & Related papers  (2023-09-17T12:06:04Z)
• DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion [144.9653045465908]
  We propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM) Our approach yields promising fusion results in infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2023-03-13T04:06:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.