Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

• URL: http://arxiv.org/abs/2607.12397v1
• Date: Tue, 14 Jul 2026 06:17:09 GMT
• Title: Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents
• Abstract summary: LLM agents act in external environments where each action changes the state that later decisions condition on, and where a single wrong step can waste interaction budget or trigger irreversible side effects long before the final failure is observed.<n>We introduce the method (methodshort), a self-evolving critic framework in which an LLM critic accumulates evidence from its own past judgments and their observed consequences.
• Score: 7.774009529494805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents act in external environments where each action changes the state that later decisions condition on, and where a single wrong step can waste interaction budget or trigger irreversible side effects long before the final failure is observed. Reliable deployment therefore requires \emph{step-level confidence estimation}: a calibrated probability that each proposed action is productive, available \emph{before} the action is executed. Existing LLM confidence estimators are designed to score a response from the given prompt, but agent confidence also depends on execution consequences: whether similar actions in similar situations actually advanced the task after the environment responded. We introduce the \method (\methodshort), a self-evolving critic framework in which an LLM critic accumulates evidence from its own past judgments and their observed consequences. After each trajectory, a hindsight LLM that sees the full execution feedback votes on whether each step was productive. The resulting pseudo-labels populate a memory bank from which related productive and unproductive experiences are retrieved into the critic's prompt whenever a similar step recurs. \methodshort requires no training and uses no ground truth step labels. Across three agent benchmarks and three critic backbones, \methodshort attains the best calibration (ECE and Brier) and ranking (AUC) in every dataset--critic combination, reducing ECE by up to $54\%$ relative to the strongest training-free baseline.

Related papers

• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? [66.17176838165994]
  Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience.<n>It remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions.<n>We introduce textbfStextsuperscript3Gym, an interactive benchmark for evaluating LLM self-improvement.
  arXiv  Detail & Related papers  (2026-08-31T17:05:41Z)
• Can LLM Rerankers Predict Their Own Ranking Performance? [73.21268576961573]
  We study textitreranker-internal QPP: can an LLM reranker estimate the quality of the ranking it has just produced?<n>For training-free estimation, we examine metric-specific self-consistency across sampled rankings and verbalized confidence produced directly by the reranker.<n>Experiments on TREC Deep Learning 2019--2022 with four LLMs show that self-consistency is competitive with the state-of-the-art (SOTA) approach and better calibrated in almost all settings.<n>To improve verbalized confidence, we propose two supervised methods, Verb
  arXiv  Detail & Related papers  (2026-06-02T11:57:27Z)
• BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents [24.997265534888626]
  We present textbfBenchTrace, a benchmark for evaluating self-evolution ability in LLM agents.<n>BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks.<n>We propose textbffailure avoidance rate (FAR), a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance.
  arXiv  Detail & Related papers  (2026-05-28T01:25:37Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation [2.102846336724103]
  Procedure-Aware Evaluation (PAE) is a framework that formalizes agent procedures as structured observations.<n>We evaluate state-of-the-art Large Language Model (LLM)-based agents on tau-bench.
  arXiv  Detail & Related papers  (2026-03-03T15:47:41Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detection [1.1087735229999818]
  Self-evaluating large language models (LLMs) provides meaningful confidence estimates in complex reasoning.<n>Stepwise evaluation generally outperforms holistic scoring in detecting potential errors.
  arXiv  Detail & Related papers  (2025-11-10T18:19:51Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.