LACE: Loss-Adaptive Capacity Expansion for Continual Learning

• URL: http://arxiv.org/abs/2603.28611v1
• Date: Mon, 30 Mar 2026 15:58:33 GMT
• Title: LACE: Loss-Adaptive Capacity Expansion for Continual Learning
• Abstract summary: LACE (Loss-Adaptive Capacity Expansion) is an online mechanism that expands a model's representational capacity during training.<n>LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.

Related papers

• Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension [13.776610961556202]
  We introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients.<n>By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability.<n> Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment.
  arXiv  Detail & Related papers  (2026-07-27T12:46:19Z)
• SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models [75.75091459417304]
  Large Vision-Language Models (LVLMs) have achieved remarkable progress in multimodal understanding.<n>LVLMs incur substantial memory and latency overhead, severely limiting real-world deployment on resource-constrained devices.<n>We propose a binarization method to drastically reduce storage and computational costs.
  arXiv  Detail & Related papers  (2026-07-02T08:30:00Z)
• Scaling Adaptive Depth with Norm-Agnostic Residual Networks [9.219329760729629]
  NAG is a norm-agnostic residual architecture that separates magnitude from directional information in the residual stream.<n>NAG introduces only a negligible number of additional parameters and relies on simple operations that are easily kernel-fusible.<n>We show that this architecture outperforms baseline Transformers, with gains that increase substantially as depth grows.
  arXiv  Detail & Related papers  (2026-06-15T02:05:37Z)
• SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere [90.85001795656633]
  Video anomaly detection (VAD) aims to automatically identify events that deviate from normal patterns in untrimmed surveillance videos.<n>We propose SphereVAD, a fully training-free, zero-shot VAD framework that recasts anomaly discrimination as von Mises-Fisher (vMF) likelihood-ratio geodesic inference on the unit hypersphere.
  arXiv  Detail & Related papers  (2026-05-08T16:57:38Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation [69.7036779439312]
  Black box domain adaptation represents a highly practical yet extremely challenging setting.<n>We propose a dual teacher distillation with subnetwork rectification (DDSR) model that exploits specific knowledge embedded in black box source models.<n>Our approach demonstrates consistent improvements over state of the art methods, including those using source data or models.
  arXiv  Detail & Related papers  (2026-03-24T07:54:19Z)
• The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation [86.83038620308423]
  Continual Test-Time Adaptation (CTTA) aims to enable models to adapt online to unlabeled data streams under distribution shift without accessing source data.<n>We propose Guided Online Low-rank Directional adaptation (GOLD), which uses a lightweight adapter to project features onto the golden subspace and learns a compact scaling vector while the subspace is dynamically updated via AGOP.
  arXiv  Detail & Related papers  (2026-03-23T12:48:38Z)
• Efficient Learning of Sparse Representations from Interactions [9.381985901356922]
  We propose a training strategy for learning high-dimensional sparse embedding layers in place of conventional dense ones.<n>We modified the production-grade collaborative filtering autoencoder ELSA, achieving up to 10x reduction in embedding size with no loss of recommendation accuracy.
  arXiv  Detail & Related papers  (2026-02-10T16:09:58Z)
• FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning [24.195588298488314]
  Existing methods suppress parameters or confuse knowledge without explicit constraints on both feature and gradient level.<n>We propose FG-OrIU (textbfFeaturetextbfGradient textbfOrthogonality for textbfIncrementaltextbfUnlearning)<n>It decomposes feature spaces via Singular Value Decomposition (SVD), separating forgetting and remaining class features into distinct subspaces.
  arXiv  Detail & Related papers  (2026-01-20T04:05:13Z)
• ELLA: Efficient Lifelong Learning for Adapters in Large Language Models [12.489255789379817]
  Large Language Models (LLMs) suffer severe catastrophic forgetting when adapted sequentially to new tasks in a continual learning setting.<n>We introduce ELLA, a training framework built on the principle of selective subspace de-correlation.<n>ELLA explicitly characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions.<n>It achieves state-of-the-art CL performance on three popular benchmarks, with relative accuracy gains of up to $9.6%$ and a $35times$ smaller memory footprint.
  arXiv  Detail & Related papers  (2026-01-05T15:58:08Z)
• Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learning [73.59672160329296]
  CKPD-FSCIL is a unified framework that unlocks the underutilized capacity of pretrained weights.<n>Our method consistently outperforms state-of-the-art approaches in both adaptability and knowledge retention.
  arXiv  Detail & Related papers  (2025-01-09T07:18:48Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.