Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations

• URL: http://arxiv.org/abs/2609.09448v1
• Date: Tue, 08 Sep 2026 20:58:49 GMT
• Title: Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
• Abstract summary: In comparison to the traditional machine learning systems, agentic have complex failure modes with planning, tool invocation and dynamic environment interactions.<n>We investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups.
• Score: 0.09085204695117637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As agentic systems getting adopted rapidly in safety critical applications, it is vital to measure the confidence associated with the agentic actions. In comparison to the traditional machine learning systems, agentic workflows have complex failure modes with planning, tool invocation and dynamic environment interactions. In this paper, we investigate whether model's internal representations provide stronger signals of eventual task success in multi-turn agentic setups. We introduce two complementary methods: Latent Trajectory Dynamics (LTD), which summarizes changes in residual-stream representations across an an interaction trajectory, and the Action Representation Probe (ARP), which predicts success from representations formed at action decisions. Across three interactive benchmarks (Bash, SQL, Python) and three model families (Qwen14B, Qwen7B, DeepSeek6.7B), our methods consistently outperform surface level generation and sequence-based calibration baselines providing a zero-overhead reliability monitor that requires neither prompt alterations nor multi-sample rollouts.

Related papers

• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Agent Drift: Quantifying Behavioral Degradation in Multi-Agent LLM Systems Over Extended Interactions [0.0]
  Agent drift is the progressive degradation of agent behavior, decision quality, and inter-agent coherence over extended interaction sequences.<n>We introduce the Agent Stability Index (ASI), a novel composite metric for quantifying drift across twelve dimensions.<n>We show how unchecked agent drift can lead to substantial reductions in task completion accuracy and increased human intervention requirements.
  arXiv  Detail & Related papers  (2026-01-07T18:37:26Z)
• Convergence dynamics of Agent-to-Agent Interactions with Misaligned objectives [3.9351446512514947]
  We develop a theoretical framework for agent-to-agent interactions in multi-agent scenarios.<n>We characterize the generation dynamics associated with the interaction when the agents have misaligned objectives.<n>Our framework presents a setup to study, predict, and defend multi-agent systems.
  arXiv  Detail & Related papers  (2025-11-11T19:18:55Z)
• Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigm [60.36837655498119]
  We propose a Trajectory-based validated-by-Reproducing Agent-benchmark Complexity Evolution framework.<n>This framework takes an original task from an existing benchmark and encourages agents to evolve it into a new task with higher difficulty.<n>Experiments on the GAIA benchmark demonstrate that the TRACE framework consistently enhances task complexity while improving the reliability of correctness.
  arXiv  Detail & Related papers  (2025-10-01T01:52:52Z)
• AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production [4.031479494871582]
  We present Agent, the first evaluation framework designed specifically for post-deployment monitoring and reasoning of agentic pipeline.<n>Agent achieves state-of-the-art results on key metrics, while uncovering critical issues missed in human annotations.
  arXiv  Detail & Related papers  (2025-09-18T05:59:04Z)
• AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models [23.916663925674737]
  Previous work has shown that current LLM-based agents execute numerous malicious tasks even without being attacked.<n>We propose AgentAlign, a novel framework that leverages abstract behavior chains as a medium for safety alignment data synthesis.<n>Our framework enables the generation of highly authentic and executable instructions while capturing complex multi-step dynamics.
  arXiv  Detail & Related papers  (2025-05-29T03:02:18Z)
• SrSv: Integrating Sequential Rollouts with Sequential Value Estimation for Multi-agent Reinforcement Learning [23.032729815716813]
  High complexity of real-world environments exacerbates the credit assignment problem.<n>The variability of agent populations in large-scale scenarios necessitates scalable decision-making mechanisms.<n>We propose a novel framework: Sequential rollout with Sequential value estimation (SrSv)
  arXiv  Detail & Related papers  (2025-03-03T12:17:18Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning [84.22561239481901]
  We propose a new approach that enables agents to learn whether their behaviors should be consistent with that of other agents. We evaluate DCIR in multiple environments including Multi-agent Particle, Google Research Football and StarCraft II Micromanagement.
  arXiv  Detail & Related papers  (2023-12-10T06:03:57Z)
• Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation [58.21683603243387]
  We propose three auxiliary tasks with relational-temporal reasoning and integrate them into the standard Deep Learning framework. These auxiliary tasks provide additional supervision signals to infer the behavior patterns other interactive agents. Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics.
  arXiv  Detail & Related papers  (2023-11-27T18:57:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.