Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning
Abstract Overview
Janus-LoRA is a continual learning framework built on LoRA that targets the stability-plasticity trade-off from two levels: parameter updates and feature geometry. The paper argues that standard LoRA optimization violates the intended orthogonality to previously learned task subspaces because its low-rank factors are updated independently, which can reintroduce interference and forgetting. To address this, the method combines Gradient Rectification, which maps a projected safe full-weight update into rectified LoRA factor updates, with an Online Estimation procedure that tracks the protected historical subspace without storing past data. It also adds a Decoupled Margin Loss to push new features away from historical class prototypes, aiming to reduce feature-space encroachment while retaining adaptability to new tasks.
Novelty
The paper’s main novelty is its diagnosis that catastrophic forgetting in LoRA-based continual learning arises not only from insufficient orthogonality constraints, but from an internal mismatch between the desired safe full-weight update and the actual composite update produced by independently optimized LoRA factors. It introduces a closed-form Gradient Rectification step, coupled with online subspace estimation and a feature-level Decoupled Margin Loss, to jointly enforce parameter-level orthogonality and feature-level separation.
Results
Across the reported class-incremental benchmarks, Janus-LoRA attains the best ACC and MAA values among the compared methods under the authors’ unified implementation protocol, including ImageNet-R, CIFAR-100, ImageNet-100, and DomainNet. On ImageNet-R with 20 tasks, it reports 71.57 ACC and 77.11 MAA, exceeding the next-best MAA of 76.24 from InfLoRA; on CIFAR-100 it reports 88.68 ACC and 92.58 MAA. Ablation results on ImageNet-R also show that combining Online Estimation and Gradient Rectification sharply reduces forgetting (best BWT -4.43 in the ablation table), while the full model gives the highest overall ACC and MAA.
Key Points
- Janus-LoRA combines three components—Online Estimation, Gradient Rectification, and Decoupled Margin Loss—to balance forgetting prevention with learning of new tasks in exemplar-free continual learning.
- The method is motivated by a specific failure mode of standard LoRA: independent updates to the low-rank factors can make the resulting full-weight update deviate from the orthogonal safe direction intended to protect prior knowledge.
- Empirical results and ablations indicate that parameter-level correction is central for stability, while the feature-level margin loss improves overall accuracy by creating more separated regions for new representations.