BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

• URL: http://arxiv.org/abs/2605.29225v1
• Date: Thu, 28 May 2026 01:25:37 GMT
• Title: BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents
• Abstract summary: We present textbfBenchTrace, a benchmark for evaluating self-evolution ability in LLM agents.<n>BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks.<n>We propose textbffailure avoidance rate (FAR), a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance.
• Score: 24.997265534888626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, leaving reflection quality unknown, and it relies on agents' own episode runs, offering no mechanism to target specific failure patterns. We present \textbf{BenchTrace}, a benchmark for evaluating self-evolution ability in LLM agents. BenchTrace is built on a snapshot-reflection dataset of 1,821 annotated episodes spanning six diverse tasks, and comprises a \textbf{Reflection Evaluation} that probes failure identification through targeted QA tasks, and an \textbf{Evolution Evaluation} that tests whether past failure experience translates into avoidance behavior in a controlled self-evolution simulation. Building on BenchTrace, we propose \textbf{failure avoidance rate (FAR)}, a new evaluation metric measuring the fraction of test cases in which the agent successfully avoids the target failure instance. Experiments with Qwen3-32B and GPT-4.1 reveal that both models fall below a 30\% end-to-end pass rate on reflection evaluation, with diagnosis as the primary bottleneck. Evolution evaluation shows that self-evolution methods generally improve FAR over the non-evolving baseline, but agents forget early lessons as noise episodes accumulate, and agents fail to generalize their reflections beyond the specific context, causing negative transfer across task contexts. Our correlation analysis further reveals that only a fully correct reflection is strongly associated with higher FAR. BenchTrace exposes concrete limits of current self-evolution approaches and provides a controlled, model-agnostic framework for targeted evaluation.

Related papers

• HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution [11.431479238875712]
  We introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution.<n>We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks.<n>Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings.
  arXiv  Detail & Related papers  (2026-09-01T07:31:18Z)
• Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration [14.41671360589919]
  DoCtOR (Diagnose-then-Correct PPO-enhanced Reflection) is a novel reflection framework that enhances multi-agent collaboration.<n>DoCtOR achieves 22%, 26%, and 27% improvements over initial success rates on HotPotQA, ChartQAPro, and Mind2Web datasets.
  arXiv  Detail & Related papers  (2026-08-28T12:26:03Z)
• Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds [7.692886317666416]
  Large language models (LLMs) are increasingly prompted to "reflect," yet whether this resembles human revision remains unclear.<n>We introduce the Human-LLM Reflection Framework (HRF), a controlled two-pass protocol comparing human and LLM revision.<n>On objective tasks, reflection yields near-zero information gain (Delta I approx 0), behaving as neutral re-generation indistinguishable from re-sampling.<n>On subjective tasks, it yields significant negative gain (Delta I 0), moving predictions away from the target.
  arXiv  Detail & Related papers  (2026-07-31T00:15:32Z)
• Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents [7.774009529494805]
  LLM agents act in external environments where each action changes the state that later decisions condition on, and where a single wrong step can waste interaction budget or trigger irreversible side effects long before the final failure is observed.<n>We introduce the method (methodshort), a self-evolving critic framework in which an LLM critic accumulates evidence from its own past judgments and their observed consequences.
  arXiv  Detail & Related papers  (2026-07-14T06:17:09Z)
• Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection [80.15317858033534]
  ForeAgent is an agentic forensics framework for AI-generated image detection with iterative self-evolution.<n>ForeAgent achieves state-of-the-art performance on the Chameleon benchmark, reaching 82.18% accuracy.<n>ForeAgent produces more consistent and causally grounded reasoning compared to GPT-5 and GPT-5-mini.
  arXiv  Detail & Related papers  (2026-06-25T02:59:33Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs [63.88783817420284]
  Embodied robots cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials.<n>We introduce Reflective Test-Time Planning, which integrates two modes of reflection: textitreflection-in-action and textitreflection-on-action<n>We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight.
  arXiv  Detail & Related papers  (2026-02-24T18:55:18Z)
• RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training [59.493415006017635]
  Pre-trained Multi-modal Large Language Models (MLLMs) provide a knowledge-rich foundation for post-training.<n>Current evaluation relies on testing after supervised fine-tuning, which introduces laborious additional training and autoregressive decoding costs.<n>We propose RADAR, an efficient ability-centric evaluation framework for Revealing Asymmetric Development of Abilities in MLLM pRe-training.
  arXiv  Detail & Related papers  (2026-02-13T12:56:31Z)
• Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics [0.0]
  Vision-Language Agents (VLAs) can decompose complex visual tasks into executable tool-based plans.<n>Recent benchmarks have begun to evaluate iterative self-correction, but its quantitative limits and dominant reasoning bottlenecks remain poorly characterized.
  arXiv  Detail & Related papers  (2026-01-14T15:17:11Z)
• AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering [8.201374511929538]
  AgentDevel is a release engineering pipeline that iteratively runs the current agent.<n>It produces implementation-blind, symptom-level quality signals from execution traces.<n>It aggregates dominant symptom patterns and produces auditable engineering specifications.
  arXiv  Detail & Related papers  (2026-01-08T05:49:01Z)
• Reflective Confidence: Correcting Reasoning Flaws via Online Self-Correction [14.164508061248775]
  Large language models (LLMs) have achieved strong performance on complex reasoning tasks using techniques such as chain-of-thought and self-consistency.<n>We propose reflective confidence, a novel reasoning framework that transforms low-confidence signals from termination indicators into reflection triggers.<n> Experiments on mathematical reasoning benchmarks, including AIME 2025, demonstrate significant accuracy improvements over advanced early-stopping baselines at comparable computational cost.
  arXiv  Detail & Related papers  (2025-12-21T05:35:07Z)
• Self-Reflective Generation at Test Time [42.02273611421918]
  Large language models (LLMs) increasingly solve complex reasoning tasks via long chain-of-thought.<n>Existing self-reflection either performs revisions over full drafts or learns self-correction via expensive training.<n>We propose Self-Reflective Generation at Test Time (SRGen), a lightweight test-time framework that reflects before generating at uncertain points.
  arXiv  Detail & Related papers  (2025-10-03T11:46:04Z)
• Solving Offline Reinforcement Learning with Decision Tree Regression [0.0]
  This study presents a novel approach to addressing offline reinforcement learning problems by reframing them as regression tasks. We introduce two distinct frameworks: return-conditioned and return-weighted decision tree policies. Despite the simplification inherent in this reformulated approach to offline RL, our agents demonstrate performance that is at least on par with the established methods.
  arXiv  Detail & Related papers  (2024-01-21T23:50:46Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.