Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

• URL: http://arxiv.org/abs/2603.07392v1
• Date: Sun, 08 Mar 2026 00:38:39 GMT
• Title: Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
• Abstract summary: We evaluate online adaptation over streaming, continually updating knowledge.<n> OAKS comprises two datasets: OAKS-BABI and OAKS-Novel.<n>Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS.
• Score: 78.9560820155271
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.

Related papers

• When Does Continual Learning Require Learning [68.55508958298942]
  We show that continual learning is not a single capability.<n>Different patterns of environmental change require fundamentally different update behaviors.<n>We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.
  arXiv  Detail & Related papers  (2026-07-08T18:27:40Z)
• Self-Consolidating Language Models: Continual Knowledge Incorporation from Context [5.0834716824529105]
  Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context.<n>We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information.<n>We propose textbfSelf-textbfConsolidating textbfLanguage Models (SCoL), a post-training framework in which an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated.
  arXiv  Detail & Related papers  (2026-05-08T00:50:09Z)
• LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection [11.420370070522205]
  We introduce LiveFact, a benchmark that simulates the real-world "fog of war" in misinformation detection.<n>LiveFact uses dynamic, temporal evidence sets to evaluate models on their ability to reason with evolving, incomplete information.<n>Tests show that open-source Mixture-of-Experts models, such as Qwen3-235B-A22B, now match or outperform proprietary state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-04-06T16:20:47Z)
• CLeAN: Continual Learning Adaptive Normalization in Dynamic Environments [3.4182067509295915]
  We introduce Continual Learning Adaptive Normalization (CLeAN), a novel adaptive normalization technique for continuous learning.<n> CLeAN involves the estimation of global feature scales using learnable parameters that are updated via an Exponential Moving Average (EMA) module.<n>We demonstrate that CLeAN not only improves model performance on new data but also mitigates catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-03-18T09:52:31Z)
• V-DyKnow: A Dynamic Benchmark for Time-Sensitive Knowledge in Vision Language Models [1.424507155580441]
  Real-world facts are intrinsically time-sensitive and subject to erratic and periodic changes.<n>We present V-DyKnow, a benchmark for evaluating time-sensitive factual knowledge in Vision-Language Models.
  arXiv  Detail & Related papers  (2026-03-17T14:33:08Z)
• Experience Scaling: Post-Deployment Evolution For Large Language Models [44.48142891798125]
  We propose experience scaling, a framework for continuous post-deployment evolution for large language models (LLMs)<n>We validate the framework in simulated real-world scenarios involving generalization to previously unseen but related tasks, repetitive queries, and over-saturated knowledge stores.<n>Results demonstrate that structured post-deployment learning can extend LLM capabilities beyond the limits of static human-generated data.
  arXiv  Detail & Related papers  (2025-09-23T08:04:58Z)
• In-context learning of evolving data streams with tabular foundational models [42.13420474990124]
  This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments.<n> Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees across all non-stationary benchmarks.
  arXiv  Detail & Related papers  (2025-02-24T04:52:35Z)
• ODEStream: A Buffer-Free Online Learning Framework with ODE-based Adaptor for Streaming Time Series Forecasting [11.261457967759688]
  ODEStream is a buffer-free continual learning framework that incorporates a temporal isolation layer to capture temporal dependencies within the data.<n>It generates a continuous data representation, enabling seamless adaptation to changing dynamics in a data streaming scenario.<n>Our approach focuses on learning how the dynamics and distribution of historical data change over time, facilitating direct processing of streaming sequences.
  arXiv  Detail & Related papers  (2024-11-11T22:36:33Z)
• A Practitioner's Guide to Continual Multimodal Pretraining [83.63894495064855]
  Multimodal foundation models serve numerous applications at the intersection of vision and language.<n>To keep models updated, research into continual pretraining mainly explores scenarios with either infrequent, indiscriminate updates on large-scale new data, or frequent, sample-level updates.<n>We introduce FoMo-in-Flux, a continual multimodal pretraining benchmark with realistic compute constraints and practical deployment requirements.
  arXiv  Detail & Related papers  (2024-08-26T17:59:01Z)
• Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models [74.81091933317882]
  We introduce EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. We uncover that existing continual learning baselines suffer from updating and removing outdated knowledge. Our work aims to model the dynamic nature of real-world information, suggesting faithful evaluations of the evolution-adaptability of language models.
  arXiv  Detail & Related papers  (2023-11-14T12:12:02Z)
• Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge [72.63368052592004]
  We study LMs' abilities to make inferences based on injected facts (or propagate those facts) We find that existing methods for updating knowledge show little propagation of injected knowledge. Yet, prepending entity definitions in an LM's context improves performance across all settings.
  arXiv  Detail & Related papers  (2023-05-02T17:59:46Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment [55.91291559442884]
  This work develops a methodology that enables data-driven methods to continuously learn and optimize in a dynamic environment. We propose to build the notion of continual learning into the modeling process of learning wireless systems. Our design is based on a novel min-max formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2020-11-16T08:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.