Compact Memory for Continual Logistic Regression

• URL: http://arxiv.org/abs/2511.09167v1
• Date: Thu, 13 Nov 2025 01:37:11 GMT
• Title: Compact Memory for Continual Logistic Regression
• Abstract summary: We present a new method to build compact memory for logistic regression.<n>Our approach can drastically improve accuracy compared to Experience Replay.<n>Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.
• Score: 45.6774409797545
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.

Related papers

• Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agents [0.0]
  Large language model (LLM) agents operate over long, multi-session interactions in which facts change.<n>We show that acting correctly requires using the current value of a fact and discarding values that have been superseded.<n>We release Supersede, an open reinforcement-learning environment that turns this measurement into a training signal.
  arXiv  Detail & Related papers  (2026-06-25T18:50:32Z)
• KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning [8.216400469571084]
  We propose KEEP, a KV-cache-centric memory management system for efficient embodied planning.<n>KEEP features 3 key innovations: (1) a Static-Dynamic Memory Construction algorithm that reduces KV cache recomputation by mixed-granularity memory group; (2) a Multi-hop Memory Re-computation algorithm that dynamically identifies important cross-attention among different memory groups; and (3) a Layer-balanced Memory Loading that eliminates unbalanced KV cache loading and cross-attention across different layers.
  arXiv  Detail & Related papers  (2026-02-27T01:48:07Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Memory-Efficient Point Cloud Registration via Overlapping Region Sampling [0.7806681790324609]
  Recent advances in deep learning have improved 3D point cloud registration but increased graphics processing unit memory usage. We propose an overlapping region sampling method to reduce memory usage while maintaining accuracy. Our approach estimates the overlapping region and intensively samples from it, using a k-nearest-neighbor (kNN) based point compression mechanism.
  arXiv  Detail & Related papers  (2024-10-29T05:40:03Z)
• TEAL: New Selection Strategy for Small Buffers in Experience Replay Class Incremental Learning [7.627299398469962]
  We introduce TEAL, a novel approach to populate the memory with exemplars.<n>We show that TEAL enhances the average accuracy of existing class-incremental methods.
  arXiv  Detail & Related papers  (2024-06-30T12:09:08Z)
• Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment [1.9055921262476347]
  This paper proposes an optimization of an existing Deep Neural Network (DNN) that improves its hardware utilization and facilitates on-device training for resource-constrained edge environments. We implement efficient parameter reduction strategies on Xception that shrink the model size without sacrificing accuracy, thus decreasing memory utilization during training.
  arXiv  Detail & Related papers  (2024-03-14T19:40:58Z)
• An Evaluation of Memory Optimization Methods for Training Neural Networks [12.534553433992606]
  Development of memory optimization methods (MOMs) has emerged as a solution to address the memory bottleneck encountered when training large models. To examine the practical value of various MOMs, we have conducted a thorough analysis of existing literature from a systems perspective. Our analysis has revealed a notable challenge within the research community: the absence of standardized metrics for effectively evaluating the efficacy of MOMs.
  arXiv  Detail & Related papers  (2023-03-26T05:40:35Z)
• Calibrating the Rigged Lottery: Making All Tickets Reliable [14.353428281239665]
  We propose a new sparse training method to produce sparse models with improved confidence calibration. Our method simultaneously maintains or even improves accuracy with only a slight increase in computation and storage burden.
  arXiv  Detail & Related papers  (2023-02-18T15:53:55Z)
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning [56.450090618578]
  Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. We show that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work. We propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel.
  arXiv  Detail & Related papers  (2022-05-26T08:24:01Z)
• Memory Replay with Data Compression for Continual Learning [80.95444077825852]
  We propose memory replay with data compression to reduce the storage cost of old training samples. We extensively validate this across several benchmarks of class-incremental learning and in a realistic scenario of object detection for autonomous driving.
  arXiv  Detail & Related papers  (2022-02-14T10:26:23Z)
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
  We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
  arXiv  Detail & Related papers  (2021-10-28T17:58:45Z)
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory [75.65949969000596]
  Episodic and semantic memory are critical components of the human memory model. We develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation.
  arXiv  Detail & Related papers  (2021-02-20T18:40:40Z)
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning [78.80707950262214]
  On-device learning enables edge devices to continually adapt the AI models to new data. Existing work solves this problem by reducing the number of trainable parameters. We present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning.
  arXiv  Detail & Related papers  (2020-07-22T18:39:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.