Abstraction as a Memory-Efficient Inductive Bias for Continual Learning

• URL: http://arxiv.org/abs/2603.17198v1
• Date: Tue, 17 Mar 2026 22:59:13 GMT
• Title: Abstraction as a Memory-Efficient Inductive Bias for Continual Learning
• Abstract summary: Abstraction-Augmented Training (AAT) is a loss-level modification encouraging models to capture the latent relational structure shared across examples.<n>AAT stabilizes learning in strictly online data streams, eliminating the need for a replay buffer.<n>Our results show that AAT achieves performance comparable to or exceeding strong experience replay (ER) baselines.
• Score: 2.8602273445853967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The real world is non-stationary and infinitely complex, requiring intelligent agents to learn continually without the prohibitive cost of retraining from scratch. While online continual learning offers a framework for this setting, learning new information often interferes with previously acquired knowledge, causes forgetting and degraded generalization. To address this, we propose Abstraction-Augmented Training (AAT), a loss-level modification encouraging models to capture the latent relational structure shared across examples. By jointly optimizing over concrete instances and their abstract representations, AAT introduces a memory-efficient inductive bias that stabilizes learning in strictly online data streams, eliminating the need for a replay buffer. To capture the multi-faceted nature of abstraction, we introduce and evaluate AAT on two benchmarks: a controlled relational dataset where abstraction is realized through entity masking, and a narrative dataset where abstraction is expressed through shared proverbs. Our results show that AAT achieves performance comparable to or exceeding strong experience replay (ER) baselines, despite requiring zero additional memory and only minimal changes to the training objective. This work highlights structural abstraction as a powerful, memory-free alternative to ER.

Related papers

• On-the-go Forgetting without Explicit Unlearning via ERASE [2.6426981014417046]
  ERASE is a framework for on-the-go forgetting that suppresses the observable influence of private data without modifying model weights.<n>We rigorously characterize sufficient conditions when ERASE provably achieves functional forgetting of designated subclasses.<n> ERASE maintains the best observed balance between forgetting efficacy, computational efficiency, and retention fidelity over recent unlearning-based methods.
  arXiv  Detail & Related papers  (2026-09-05T08:09:27Z)
• Continual Self-Improvement with Lightweight Experiential Latent Memories [5.476321641759679]
  Large language models achieve strong reasoning performance by scaling inference-time compute.<n>We show that In-Context Learning (ICL) over raw reasoning traces fails to generalize.<n>We propose an online method that distills inference-time compute spent on encountered problems into compact modular latent memories.
  arXiv  Detail & Related papers  (2026-06-16T11:27:28Z)
• SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents [35.45207852488779]
  We introduce a Tool-Memory based self-evolving agentic framework SEARL.<n>Unlike approaches that directly utilize interaction experiences, our method constructs a structured experience memory that integrates planning with execution.<n>We evaluate our framework on knowledge reasoning and mathematics tasks, demonstrating its effectiveness in achieving more practical and efficient learning.
  arXiv  Detail & Related papers  (2026-04-09T04:38:47Z)
• ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Recovery [73.8181449261685]
  Federated Learning (FL) enables collaborative model training by sharing model updates instead of raw data, aiming to protect user privacy.<n>Recent studies reveal that these shared updates can inadvertently leak sensitive training data through gradient inversion attacks (GIAs)
  arXiv  Detail & Related papers  (2026-03-18T11:40:44Z)
• Knowledge Restoration-driven Prompt Optimization: Unlocking LLM Potential for Open-Domain Relational Triplet Extraction [4.01163688152329]
  Open mining of triplets is the foundation for structured knowledge without predefined schemas.<n>Existing methods exhibit vulnerability in ambiguity making often erroneous extraction patterns permanent.<n>We propose a Knowledge Reconstruction-driven Prompt Optimization (KRPO) framework to assist Large Language Models (LLMs) in continuously improving their extraction capabilities.
  arXiv  Detail & Related papers  (2026-01-21T14:42:13Z)
• Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning [51.07663354001582]
  Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task.<n>We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches.<n>We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods.
  arXiv  Detail & Related papers  (2025-12-01T15:56:00Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Remember and Recall: Associative-Memory-based Trajectory Prediction [25.349986959111757]
  We propose the Fragmented-Memory-based Trajectory Prediction (FMTP) model, inspired by the remarkable learning capabilities of humans. The FMTP model employs discrete representations to enhance computational efficiency by reducing information redundancy. We develop an advanced reasoning engine based on language models to deeply learn the associative rules among these discrete representations.
  arXiv  Detail & Related papers  (2024-10-03T04:32:21Z)
• DELTA: Dynamic Embedding Learning with Truncated Conscious Attention for CTR Prediction [61.68415731896613]
  Click-Through Rate (CTR) prediction is a pivotal task in product and content recommendation. We propose a model that enables Dynamic Embedding Learning with Truncated Conscious Attention for CTR prediction.
  arXiv  Detail & Related papers  (2023-05-03T12:34:45Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)
• Online Continual Learning on Sequences [9.603184477806954]
  Online continual learning (OCL) refers to the ability of a system to learn over time from a continuous stream of data without having to revisit previously encountered training samples. Machine learning models that address OCL must alleviate textitcatastrophic forgetting in which hidden representations are disrupted or completely overwritten when learning from streams of novel input.
  arXiv  Detail & Related papers  (2020-03-20T05:49:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.