TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning

• URL: http://arxiv.org/abs/2606.06494v1
• Date: Thu, 04 Jun 2026 17:59:55 GMT
• Title: TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
• Abstract summary: We introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix.<n>A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.
• Score: 39.35080737558648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in Continual Learning. In this paper we introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.

Related papers

• Principal-timestep Restricted Init via Sparse Matrix-decomposition in Flow-matching [17.26864820077326]
  We propose Prism-LoRA, a Principal-timestep Restricted Init via Sparse Matrix-decomposition framework that improves gradient alignment during fine-tuning.<n>Our method consistently improves convergence speed and final performance across multiple diffusion fine-tuning benchmarks.
  arXiv  Detail & Related papers  (2026-09-14T14:30:54Z)
• Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions [17.58096703161218]
  Analytic Continual Learning offers a computationally efficient alternative to gradient-based approaches.<n>Recent ACL methods falter significantly in Class-Incremental Learning scenarios characterized by Long-Tailed distributions.<n>We propose Geometry-Spectral Rectification (GSR), a theoretically grounded framework that treats long-tailed learning as a spectral regularization problem.
  arXiv  Detail & Related papers  (2026-07-24T22:12:15Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Regularized Centered Emphatic Temporal Difference Learning [14.415762410365106]
  Off-policy temporal-difference (TD) learning with function approximation faces a structural tradeoff among stability, projection geometry, and variance control.<n>Emphatic TD (ETD) improves the off-policy projection geometry through follow-on emphasis, but the follow-on trace can have high variance.<n>We propose emphRegularized Emphatic Temporal-Difference Learning (RETD), which preserves the follow-on trace and regularizes only the auxiliary centering recursion.
  arXiv  Detail & Related papers  (2026-05-02T10:32:24Z)
• On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning [14.815508281465272]
  We show that forgetting is strongly influenced by the geometry and parameterization of the update subspace.<n>Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.
  arXiv  Detail & Related papers  (2026-03-10T13:53:25Z)
• Predictive Spectral Calibration for Source-Free Test-Time Regression [0.764671395172401]
  Predictive Spectral (PSC) is a source-free framework that extends subspace alignment to block spectral matching.<n> PSC remains simple to implement, model-agnostic, and compatible with off-the-shelf pretrained regressors.
  arXiv  Detail & Related papers  (2026-03-10T08:15:44Z)
• When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning [3.8218584696400484]
  Privacy-preserving training on sensitive data relies on gradient clipping and Gaussian noise.<n>Existing adaptive clipping methods often depend on per-example gradient norm statistics.<n>We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from parameters.
  arXiv  Detail & Related papers  (2026-02-11T07:18:25Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Adaptive Benign Overfitting (ABO): Overparameterized RLS for Online Learning in Non-stationary Time-series [0.0]
  ABO is highly accurate (comparable to baseline kernel methods) while achieving speed improvements of between 20 and 40 percent.<n>Results provide a unified view linking adaptive filtering, kernel approximation, and benign overfitting within a stable online learning framework.
  arXiv  Detail & Related papers  (2026-01-29T15:58:01Z)
• Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
  We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
  arXiv  Detail & Related papers  (2021-03-07T00:15:44Z)
• On Learning Rates and Schr\"odinger Operators [105.32118775014015]
  We present a general theoretical analysis of the effect of the learning rate. We find that the learning rate tends to zero for a broad non- neural class functions.
  arXiv  Detail & Related papers  (2020-04-15T09:52:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.