Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents

• URL: http://arxiv.org/abs/2608.29685v1
• Date: Sun, 30 Aug 2026 09:43:08 GMT
• Title: Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents
• Abstract summary: Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs.<n>We evaluate mainstream uncertainty signals on deep-research tasks and find that verbal confidence reliably distinguishes failures at trajectory completion.<n>We identify an underlying mechanism explaining this gap: path switching, where agents frequently abandon their current search direction in-trajectory, breaking the link between early signal and final outcome.
• Score: 13.560515796350046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs. Uncertainty quantification, such as verbal confidence and perplexity, offers a promising approach to detecting agent failures; however, it has not been explored whether these signals retain their discriminative power during the intermediate stages of long-horizon execution. We evaluate mainstream uncertainty signals on deep-research tasks and find that verbal confidence reliably distinguishes failures at trajectory completion, achieving a mean AUROC of 0.85, whereas all evaluated signals offer limited predictive value earlier in execution, with none exceeding a mean AUROC of 0.60 at 50% trajectory progress. We identify an underlying mechanism explaining this gap: path switching, where agents frequently abandon their current search direction in-trajectory, breaking the link between early signal and final outcome. These findings challenge the assumption that intermediate uncertainty can reliably guide early intervention. They also motivate a practical recommendation for agent harnesses in deep-research settings: use final-step confidence to decide whether to restart, an approach that our experiments find more effective than in-trajectory intervention.

Related papers

• Beyond End-to-End Success: Diagnosing Failures in Long-Horizon Security LLM Agents [8.355245794476078]
  We present a diagnostic methodology that instruments security tasks with checkpoints, separates failures before and after capability exposure, and uses controlled interventions to test suspected upstream bottlenecks.<n>We evaluate the methodology across four task families involving delayed reuse of discovered information, reuse of observed state, recovery from failed strategies, and decision making after uncertain outcomes.
  arXiv  Detail & Related papers  (2026-08-20T20:55:22Z)
• When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories [20.332123003765837]
  Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail.<n>This is challenging because supervision is typically available only as a trajectory-level success/failure label while alerts must be raised from partial interactions.<n>We introduce a two-stage approach that learns from this sparse evidence structure and uses the resulting risk estimates for controllable early alerting.
  arXiv  Detail & Related papers  (2026-06-03T20:28:27Z)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling [57.42714978834704]
  ExComm is a communication protocol for exploration-stage agentic test-time scaling.<n>We show that ExComm consistently outperforms strong test-time scaling baselines.
  arXiv  Detail & Related papers  (2026-05-21T07:38:44Z)
• Towards Generation-Efficient Uncertainty Estimation in Large Language Models [6.111244917760012]
  Uncertainty estimation is important for deploying LLMs in high-stakes applications such as healthcare and finance.<n>Existing methods require one or more full autoregressive generations to estimate uncertainty.<n>We investigate whether effective uncertainty estimation can be achieved with partial generation or even input-only information.
  arXiv  Detail & Related papers  (2026-05-07T11:39:13Z)
• Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization [41.15414881730464]
  Vision-Language Models (VLMs) offer a general perceive-reason-act framework for this goal.<n>Previous approaches rely on inefficient and often inaccurate implicit learning of state-values from noisy foresight predictions.<n>We propose a novel test-time computation framework that decouples state evaluation from action generation.
  arXiv  Detail & Related papers  (2026-02-22T22:53:16Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Pedestrian Trajectory Forecasting Using Deep Ensembles Under Sensing Uncertainty [125.41260574344933]
  We consider an encoder-decoder based deep ensemble network for capturing both perception and predictive uncertainty simultaneously. Overall, deep ensembles provided more robust predictions and the consideration of upstream uncertainty further increased the estimation accuracy for the model.
  arXiv  Detail & Related papers  (2023-05-26T04:27:48Z)
• Unsupervised Anomaly Detection with Rejection [19.136286864839846]
  Anomaly detectors learn a decision boundary by employing intuitions, which are hard to verify in practice. A way to combat this is by allowing the detector to reject examples with high uncertainty. This requires employing a confidence metric that captures the distance to the decision boundary and setting a rejection threshold to reject low-confidence predictions.
  arXiv  Detail & Related papers  (2023-05-22T16:22:32Z)
• Toward Reliable Human Pose Forecasting with Uncertainty [51.628234388046195]
  We develop an open-source library for human pose forecasting, including multiple models, supporting several datasets. We devise two types of uncertainty in the problem to increase performance and convey better trust.
  arXiv  Detail & Related papers  (2023-04-13T17:56:08Z)
• Temporal Difference Uncertainties as a Signal for Exploration [76.6341354269013]
  An effective approach to exploration in reinforcement learning is to rely on an agent's uncertainty over the optimal policy. In this paper, we highlight that value estimates are easily biased and temporally inconsistent. We propose a novel method for estimating uncertainty over the value function that relies on inducing a distribution over temporal difference errors.
  arXiv  Detail & Related papers  (2020-10-05T18:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.