Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

• URL: http://arxiv.org/abs/2606.05661v1
• Date: Thu, 04 Jun 2026 03:43:28 GMT
• Title: Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
• Abstract summary: We introduce Continual Learning Bench (CL-Bench), the first benchmark to measure whether AI systems genuinely improve with experience.<n> CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting)<n>We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities.
• Score: 44.90458129179607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.

Related papers

• MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems [29.473672174276743]
  We propose a user feedback simulation framework and a benchmark to evaluate the continual learning abilities of LLMsys.<n> Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying.
  arXiv  Detail & Related papers  (2025-10-20T08:16:12Z)
• ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning [73.35191368656224]
  We present textitEmbodied Reasoning Agent (ERA), a framework that integrates prior knowledge learning and online reinforcement learning.<n>ERA offers a practical path toward scalable embodied intelligence, providing methodological insights for future embodied AI systems.
  arXiv  Detail & Related papers  (2025-10-14T16:25:46Z)
• Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps [56.827895559823126]
  Self-organizing map (SOM) is a neural model often used in clustering and dimensionality reduction. We propose a generalization of the SOM, the continual SOM, which is capable of online unsupervised learning under a low memory budget. Our results, on benchmarks including MNIST, Kuzushiji-MNIST, and Fashion-MNIST, show almost a two times increase in accuracy.
  arXiv  Detail & Related papers  (2024-02-19T19:11:22Z)
• Dual Cognitive Architecture: Incorporating Biases and Multi-Memory Systems for Lifelong Learning [21.163070161951868]
  We introduce Dual Cognitive Architecture (DUCA), which includes multiple sub-systems, implicit and explicit knowledge representation, inductive bias, and a multi-memory system. DUCA shows improvement across different settings and datasets, and it also exhibits reduced task recency bias, without the need for extra information. To further test the versatility of lifelong learning methods on a challenging distribution shift, we introduce a novel domain-incremental dataset DN4IL.
  arXiv  Detail & Related papers  (2023-10-17T15:24:02Z)
• Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures [4.2903672492917755]
  We develop a biologically inspired lightweight neural network architecture that incorporates synaptic plasticity mechanisms and neuromodulation. Our approach leads to superior online continual learning performance on Split-MNIST, Split-CIFAR-10, and Split-CIFAR-100 datasets. We further demonstrate the effectiveness of our approach by integrating key design concepts into other backpropagation-based continual learning algorithms.
  arXiv  Detail & Related papers  (2023-08-08T19:12:52Z)
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems [128.63953314853327]
  "Lifelong Learning" systems are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. We show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems.
  arXiv  Detail & Related papers  (2023-01-18T21:58:54Z)
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data [101.6195176510611]
  "Online" continual learning enables evaluating both information retention and online learning efficacy. In online continual learning, each incoming small batch of data is first used for testing and then added to the training set, making the problem truly online. We introduce a new benchmark for online continual visual learning that exhibits large scale and natural distribution shifts.
  arXiv  Detail & Related papers  (2021-08-20T06:17:20Z)
• Bilevel Continual Learning [76.50127663309604]
  We present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) Our experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-30T16:00:23Z)
• Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning [4.2903672492917755]
  We develop a biologically-inspired light weight neural network architecture that incorporates local learning and neuromodulation. We demonstrate the efficacy of our approach on both single task and continual learning setting.
  arXiv  Detail & Related papers  (2020-07-16T07:41:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.