Learning, Fast and Slow: Towards LLMs That Adapt Continually

• URL: http://arxiv.org/abs/2605.12484v2
• Date: Thu, 14 May 2026 17:49:32 GMT
• Title: Learning, Fast and Slow: Towards LLMs That Adapt Continually
• Abstract summary: Large language models (LLMs) are trained for downstream tasks by updating their parameters.<n>Fast-slow training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks.
• Score: 83.9214051113102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls.

Related papers

• Attribution-Guided Continual Learning for Large Language Models [32.34727629126595]
  Large language models (LLMs) often suffer from catastrophic forgetting in continual learning.<n>Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization.<n>We propose an attribution-guided continual fine-tuning framework for LLMs.
  arXiv  Detail & Related papers  (2026-05-06T17:38:13Z)
• Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better [24.03797089794804]
  We propose a Late-to-Early Training (LET) paradigm that enables Large Language Models to learn later knowledge in earlier steps and earlier layers.<n>We identify two key mechanisms that drive LET's effectiveness: late-to-early-step learning and late-to-early-layer learning.<n>Our method achieves up to 1.6$times$ speedup with nearly 5% improvement in downstream task accuracy compared to standard training.
  arXiv  Detail & Related papers  (2026-02-05T07:19:34Z)
• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning [15.990512697643453]
  We propose a new fine-tuning algorithm termed Momentum-Filtered (MoFO)<n>MoFO achieves similar fine-tuning performance to the default fine-tuning algorithm while effectively mitigating knowledge forgetting.<n>We validate MoFO through rigorous convergence analysis and extensive experiments, demonstrating its effectiveness in mitigating forgetting without pre-training data.
  arXiv  Detail & Related papers  (2024-07-30T17:38:24Z)
• SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining [39.56934385513862]
  Training large language models (LLMs) from scratch requires significant computational power and extensive memory capacity. Recent studies have explored low-rank structures on weights for efficient fine-tuning in terms of parameters and memory. We propose to parameterize the weights as a sum of low-rank and sparse matrices for pretraining, which we call SLTrain.
  arXiv  Detail & Related papers  (2024-06-04T11:14:21Z)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs [67.38165028487242]
  We introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach to fine-tune large language models (LLMs) Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs.
  arXiv  Detail & Related papers  (2023-10-13T07:38:52Z)
• LiST: Lite Self-training Makes Efficient Few-shot Learners [91.28065455714018]
  LiST improves by 35% over classic fine-tuning methods and 6% over prompt-tuning with 96% reduction in number of trainable parameters when fine-tuned with no more than 30 labeled examples from each target domain.
  arXiv  Detail & Related papers  (2021-10-12T18:47:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.