Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

• URL: http://arxiv.org/abs/2605.10640v1
• Date: Mon, 11 May 2026 14:28:02 GMT
• Title: Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
• Abstract summary: We present a theoretical framework that characterizes the training dynamics of continual Factual Knowledge Acquisition (cFKA)<n>We propose a novel generative data replay approach, called textbfSelecting textbfTokens via attentitextbfOn textbfContribution(STOC)
• Score: 14.490693817432414
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as continual Factual Knowledge Acquisition (cFKA), remain unclear. In this work, we present a theoretical framework that characterizes the training dynamics of cFKA using a single-layer Transformer, offering a unified explanation for the behavior of representative CPT methods. Our analysis reveals that regularization-based methods merely adjust the convergence rate of parameters without altering the inherent forgetting tendency, whereas data replay methods succeed in shifting convergence dynamics and stabilizing pretrained knowledge. Building on these insights, we propose a novel generative data replay approach, called \textbf{S}electing \textbf{T}okens via attenti\textbf{O}n \textbf{C}ontribution~(STOC), which identifies influential factual snippets to guide replay data generation. Extensive experiments on both synthetic and real-world datasets validate our findings and demonstrate that STOC effectively enhances cFKA by mitigating catastrophic forgetting.

Related papers

• Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging [65.11234199720428]
  Continual Test-Time Adaptation aims to empower perception systems to handle dynamic distribution shifts encountered after deployment.<n>We introduce a novel forward-facilitation paradigm through a method termed Dynamic Style Bridging.<n>During test time, to mitigate inherent generative bias and adapt these proxies to incoming data, we propose a multi-level bridging mechanism.
  arXiv  Detail & Related papers  (2026-05-18T16:18:17Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• CIKT: A Collaborative and Iterative Knowledge Tracing Framework with Large Language Models [14.273311275013057]
  Knowledge Tracing aims to model a student's learning state over time and predict their future performance.<n>Traditional KT methods often face challenges in explainability, scalability, and effective modeling of complex knowledge dependencies.<n>We propose Collaborative Iterative Knowledge Tracing (CIKT), a framework that harnesses Large Language Models to enhance both prediction accuracy and explainability.
  arXiv  Detail & Related papers  (2025-05-23T10:16:16Z)
• AdvKT: An Adversarial Multi-Step Training Framework for Knowledge Tracing [64.79967583649407]
  Knowledge Tracing (KT) monitors students' knowledge states and simulates their responses to question sequences.<n>Existing KT models typically follow a single-step training paradigm, which leads to significant error accumulation.<n>We propose a novel Adversarial Multi-Step Training Framework for Knowledge Tracing (AdvKT) which focuses on the multi-step KT task.
  arXiv  Detail & Related papers  (2025-04-07T03:31:57Z)
• STCKGE:Continual Knowledge Graph Embedding Based on Spatial Transformation [6.534658139904185]
  We propose STCKGE, a novel Continual Knowledge Graph Embedding framework based on spatial transformation.<n>In this framework, entity positions are jointly determined by base position vectors and offset vectors.<n>We show that STCKGE's strong performance in multi-hop relationship learning and prediction accuracy, with an average MRR improvement of 5.4%.
  arXiv  Detail & Related papers  (2025-03-11T08:54:03Z)
• AAKT: Enhancing Knowledge Tracing with Alternate Autoregressive Modeling [23.247238358162157]
  Knowledge Tracing aims to predict students' future performances based on their former exercises and additional information in educational settings.<n>One of the primary challenges in autoregressive modeling for Knowledge Tracing is effectively representing the anterior (pre-response) and posterior (post-response) states of learners across exercises.<n>We propose a novel perspective on knowledge tracing task by treating it as a generative process, consistent with the principles of autoregressive models.
  arXiv  Detail & Related papers  (2025-02-17T14:09:51Z)
• Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
  CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
  arXiv  Detail & Related papers  (2024-11-18T08:20:21Z)
• Temporal-Difference Variational Continual Learning [77.92320830700797]
  We propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations.<n>Our approach effectively mitigates Catastrophic Forgetting, outperforming strong Variational CL methods.
  arXiv  Detail & Related papers  (2024-10-10T10:58:41Z)
• Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning [15.475427498268393]
  The Train-Attention-Augmented Language Model (TAALM) enhances learning efficiency by dynamically predicting and applying weights to tokens based on their usefulness.<n>We show that TAALM proves the state-of-the-art performance upon the baselines, and also shows synergistic compatibility when integrated with previous CKL approaches.
  arXiv  Detail & Related papers  (2024-07-24T01:04:34Z)
• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• History Repeats: Overcoming Catastrophic Forgetting For Event-Centric Temporal Knowledge Graph Completion [33.38304336898247]
  Temporal knowledge graph (TKG) completion models rely on having access to the entire graph during training. TKG data is often received incrementally as events unfold, leading to a dynamic non-stationary data distribution over time. We propose a general continual training framework that is applicable to any TKG completion method.
  arXiv  Detail & Related papers  (2023-05-30T01:21:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.