Exploring Data-Free LoRA Transferability for Video Diffusion Models

• URL: http://arxiv.org/abs/2605.01929v1
• Date: Sun, 03 May 2026 15:22:53 GMT
• Title: Exploring Data-Free LoRA Transferability for Video Diffusion Models
• Abstract summary: Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance.<n> adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches.<n>We propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment.
• Score: 41.981710361053196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA

Related papers

• Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy [50.63908869296697]
  Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems.<n>We propose TARA-Merging, which aligns merging weights using a preference-weighted cross-entropy pseudo-loss.<n>Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines.
  arXiv  Detail & Related papers  (2026-03-27T11:07:46Z)
• BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models [10.286339414754499]
  Bad RSSD is the first backdoor attack targeting the representation layer of self-supervised diffusion models.<n>It hijacks the semantic representations of poisoned samples with triggers in PCA space toward those of a target image.<n>Bad RSSD substantially outperforms existing attacks in both FID and MSE metrics.
  arXiv  Detail & Related papers  (2026-03-01T09:56:26Z)
• NP-LoRA: Null Space Projection Unifies Subject and Style in LoRA Fusion [16.64405108290577]
  Low-Rank Adaptation (LoRA) fusion is a key technique for reusing and composing learned subject and style representations.<n>Existing methods rely on weight-based merging, where one LoRA often dominates the other.<n>We propose Null Space Projection LoRA, a projection-based framework for LoRA fusion that enforces subspace separation.
  arXiv  Detail & Related papers  (2025-11-14T08:06:01Z)
• Latent Space Factorization in LoRA [10.994747174370099]
  Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.<n>We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA) to learn two distinct latent spaces.<n>Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
  arXiv  Detail & Related papers  (2025-10-22T14:37:20Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Frequency Domain-Based Diffusion Model for Unpaired Image Dehazing [92.61216319417208]
  We propose a novel frequency domain-based diffusion model, named ours, for fully exploiting the beneficial knowledge in unpaired clear data.<n>Inspired by the strong generative ability shown by Diffusion Models (DMs), we tackle the dehazing task from the perspective of frequency domain reconstruction.
  arXiv  Detail & Related papers  (2025-07-02T01:22:46Z)
• Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps [37.00767702936744]
  Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models.<n>Despite its successes, the substantial parameter redundancy, which limits the capacity and efficiency of LoRA, has been recognized as a bottleneck.<n>We introduce underlineSpectral-underlineencoding underlineLow-underlineRank underlineAdaptation (SeLoRA)
  arXiv  Detail & Related papers  (2025-06-20T07:09:05Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• FreSca: Scaling in Frequency Space Enhances Diffusion Models [55.75504192166779]
  This paper explores frequency-based control within latent diffusion models.<n>We introduce FreSca, a novel framework that decomposes noise difference into low- and high-frequency components.<n>FreSca operates without any model retraining or architectural change, offering model- and task-agnostic control.
  arXiv  Detail & Related papers  (2025-04-02T22:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.