Chain-of-Experience for Continual LLM Improvement

• URL: http://arxiv.org/abs/2608.18027v1
• Date: Tue, 18 Aug 2026 17:22:54 GMT
• Title: Chain-of-Experience for Continual LLM Improvement
• Abstract summary: We study how large language models learn from iterative experience at test time.<n>We instantiate Chain-of-Experience (CoE) with diverse feedback mechanisms.<n>Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines.
• Score: 57.11645184182825
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inference-time interaction. In this paper, we study how LLMs learn from iterative experience at test time, a setting we refer to as Chain-of-Experience (CoE), where models accumulate experiential traces through iterative interactions with self or environmental feedback to form a continual improvement loop beyond zero-shot inference. We instantiate CoE with diverse feedback mechanisms, including model self-feedback and environmental signals such as correctness or public coding test pass rates, and evaluate across math, coding, and knowledge domains using 8 LLMs, including GPT-5, Gemini-2.5 Pro, Claude-4.5 Sonnet. Our study shows that leveraging iterative experience consistently outperforms feedback-free baselines, achieving substantial gains with self feedback alone, alongside a 5.6% overall improvement and 19% lower API cost across tasks and models. We further show that combining complementary feedback channels (e.g., model and correctness signals) yields additional gains, and that CoE delivers higher accuracy per token than existing test-time strategies. We observe a positive correlation between LLM base ability and improvement capacity, and show that models remain robust under weak or spurious feedback, with different feedback contributing to distinct improvement aspects and most gains emerging early in the iterations.

Related papers

• Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback [8.780008955074967]
  Self-evolving large language models (LLMs) learn by generating their own training tasks and solutions, reducing reliance on human-curated supervision.<n>We propose COSE (Confidence-Orchestrated Self-Evolution), which uses the LLM's intrinsic confidence as a lightweight uncertainty signal to modulate learning.
  arXiv  Detail & Related papers  (2026-05-27T06:07:10Z)
• Experiential Reinforcement Learning [22.545003569634982]
  Experiential Reinforcement Learning (ERL) is a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process.<n>ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines.<n>These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.
  arXiv  Detail & Related papers  (2026-02-15T01:23:48Z)
• Learning from Self Critique and Refinement for Faithful LLM Summarization [26.593763749112792]
  Large Language Models (LLMs) often suffer from hallucinations when performing long-form text generation tasks such as summarization.<n>We propose Self Critique and Refinement-based Preference Optimization (SCRPO)<n>SCRPO is a self-supervised training framework that first constructs a preference dataset by leveraging the LLM's own critique and refinement capabilities.
  arXiv  Detail & Related papers  (2025-12-05T02:59:43Z)
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving [76.73222429266964]
  EvaLearn is a benchmark designed to evaluate large language models (LLMs) on their learning capability and efficiency in challenging tasks.<n>We benchmark nine frontier models and observe varied performance profiles.<n>We observe that current LLMs with stronger static abilities do not show a clear advantage in learning capability across all tasks.
  arXiv  Detail & Related papers  (2025-06-03T09:18:33Z)
• ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection [60.75785864719726]
  We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning.<n>We construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks.
  arXiv  Detail & Related papers  (2025-05-22T10:03:05Z)
• RLSR: Reinforcement Learning from Self Reward [0.0]
  We show that large language models can effectively self-improve through self-judging without reference solutions.<n>Our experiments show that models can provide reliable reward signals without ground truth answers.<n>This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning.
  arXiv  Detail & Related papers  (2025-05-12T23:51:04Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• Training Language Models to Critique With Multi-agent Feedback [102.42751835338233]
  MultiCritique pipeline improves critique ability of LLMs by utilizing multi-agent feedback. pipeline aggregates high-quality critiques from multiple agents instead of a single model. Our fine-tuned 7B model significantly surpasses other advanced 7B-13B open-source models.
  arXiv  Detail & Related papers  (2024-10-20T04:57:45Z)
• Self-Taught Evaluators [77.92610887220594]
  We present an approach that aims to im-proves without human annotations, using synthetic training data only. Our Self-Taught Evaluator can improve a strong LLM from 75.4 to 88.3 on RewardBench.
  arXiv  Detail & Related papers  (2024-08-05T17:57:02Z)
• StreamBench: Towards Benchmarking Continuous Improvement of Language Agents [63.54557575233165]
  Large language model (LLM) agents are able to improve themselves from experience, which is an important ability for continuous enhancement post-deployment. We introduce StreamBench, a benchmark designed to evaluate the continuous improvement of LLM agents over an input-feedback sequence. Our work serves as a stepping stone towards developing effective online learning strategies for LLMs, paving the way for more adaptive AI systems in streaming scenarios.
  arXiv  Detail & Related papers  (2024-06-13T02:08:28Z)
• Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems [10.58737969057445]
  We build a dataset to train a critic model capable of evaluating the citation, correctness, and fluency of responses generated by large language models. We propose an automated feedback mechanism that leverages the critic model to offer real-time feedback on heterogeneous aspects of generated text. Experimental results demonstrate the efficacy of our approach, including a 4% precision increase in citation and an approximately 8% enhancement in the MAUVE metric for fluency.
  arXiv  Detail & Related papers  (2023-09-08T09:39:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.