Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

• URL: http://arxiv.org/abs/2607.04969v1
• Date: Mon, 06 Jul 2026 11:57:32 GMT
• Title: Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
• Abstract summary: "Memorization-guided Data Reuse" is a training paradigm that adaptively determines when and how data should be reused.<n>Preliminary experiments reveal a consistent memorization-driven regime.
• Score: 11.074386515074446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's "Memorization Window" signals derived from loss retention dynamics and downstream evaluation scores, we propose "Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.

Related papers

• When to Review: Spaced Repetition for Continual Pre-Training of Language Models [1.0262304700896199]
  Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten.<n>We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step.<n>We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm.
  arXiv  Detail & Related papers  (2026-08-18T08:51:47Z)
• Incremental Evaluation and Training in Relational Deep Learning [0.0]
  We introduce an incremental, multi-episode evaluation and training paradigm to assess and improve the temporal adaptability of state-of-the-art RDL models.<n>We show that our models consistently outperform the standard, expensive, from-scratch trained baselines.
  arXiv  Detail & Related papers  (2026-08-13T09:46:29Z)
• Continual Self-Improvement with Lightweight Experiential Latent Memories [5.476321641759679]
  Large language models achieve strong reasoning performance by scaling inference-time compute.<n>We show that In-Context Learning (ICL) over raw reasoning traces fails to generalize.<n>We propose an online method that distills inference-time compute spent on encountered problems into compact modular latent memories.
  arXiv  Detail & Related papers  (2026-06-16T11:27:28Z)
• MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning [36.119948355819]
  We propose an experience replay framework that estimates sample-level memory strength and schedules rehearsal at adaptive intervals to mitigate catastrophic forgetting.<n>Extensive experiments across three backbone models and 11 sequential tasks show that MSSR consistently outperforms state-of-the-art replay baselines.
  arXiv  Detail & Related papers  (2026-03-10T16:49:44Z)
• Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data [89.96277093034547]
  We introduce EntroDrop, an entropy-guided token dropout method that functions as structured data regularization.<n>We show that EntroDrop consistently outperforms standard regularization baselines and maintains robust performance throughout extended multi-epoch training.
  arXiv  Detail & Related papers  (2025-12-29T12:35:51Z)
• Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning [51.07663354001582]
  Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task.<n>We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches.<n>We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods.
  arXiv  Detail & Related papers  (2025-12-01T15:56:00Z)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance.<n>We introduce novel algorithms for dynamic, instance-level data reweighting.<n>Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress.<n>LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset.<n>Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Adaptive Memory Replay for Continual Learning [29.333341368722653]
  Updating Foundation Models as new data becomes available can lead to catastrophic forgetting' We introduce a framework of adaptive memory replay for continual learning, where sampling of past data is phrased as a multi-armed bandit problem. We demonstrate the effectiveness of our approach, which maintains high performance while reducing forgetting by up to 10% at no training efficiency cost.
  arXiv  Detail & Related papers  (2024-04-18T22:01:56Z)
• Class-Incremental Learning Using Generative Experience Replay Based on Time-aware Regularization [24.143811670210546]
  Generative experience replay addresses the challenge of learning new tasks accumulatively without forgetting. We introduce a time-aware regularization method to fine-tune the three training objective terms used for generative replay. Experimental results indicate that our method pushes the limit of brain-inspired continual learners under such strict settings.
  arXiv  Detail & Related papers  (2023-10-05T21:07:45Z)
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening [51.34904967046097]
  Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data. We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
  arXiv  Detail & Related papers  (2023-08-15T11:30:45Z)
• Effective and Efficient Training for Sequential Recommendation using Recency Sampling [91.02268704681124]
  We propose a novel Recency-based Sampling of Sequences training objective. We show that the models enhanced with our method can achieve performances exceeding or very close to stateof-the-art BERT4Rec.
  arXiv  Detail & Related papers  (2022-07-06T13:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.