Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

• URL: http://arxiv.org/abs/2606.05145v1
• Date: Wed, 03 Jun 2026 17:50:26 GMT
• Title: Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)
• Abstract summary: We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure.<n>Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts.<n>The same three features thus convert failed traces from discarded data into a diagnostic object, supporting test-time routing and post-training analysis without training-time or weight-space access.
• Score: 25.626296075137372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional attempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods ($84.3{\pm}4.3\%$ accuracy, $+20\%$ over a majority-class baseline), and support a training-free routing rule that lifts rescue by $+12.2\%$ on the deployment-relevant Steerable-Hard subset (failures where retry is insufficient and a bounded intervention is reachable). The features and the routing rule transfer across two cross-family probes. The same three features thus convert failed traces from discarded data into a diagnostic object, supporting test-time routing and post-training analysis without training-time or weight-space access.

Related papers

• FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents [49.519184792774304]
  FailForge is an agentic framework that converts failed rollouts into training signal.<n>FailForge recovers over 26% of previously failed instances at marginal additional cost.
  arXiv  Detail & Related papers  (2026-08-09T08:22:57Z)
• TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories [48.638067300108276]
  Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure.<n>Long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context.<n>We propose Traj Debug, an error-lifecycle tracing framework that addresses long-trajectory error discovery.
  arXiv  Detail & Related papers  (2026-08-06T17:51:20Z)
• The Knowing-Saying Gap: When Probes See Errors that Confidence Misses [20.46211358172975]
  Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction.<n>Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness.<n>As a real-time monitor, probe-based interventions are sharply model and error-type dependent.
  arXiv  Detail & Related papers  (2026-07-21T12:13:10Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Tracing Agentic Failure from the Flow of Success [7.071269758592172]
  We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data.<n>We propose OAT, which casts this problem as one-class learning with neural controlled differential equations.<n>With training on only 100 successful trajectories, OAT is 200--5000 $times$ faster than prompting-based baselines.
  arXiv  Detail & Related papers  (2026-07-14T13:16:14Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability [8.036549927091286]
  This paper introduces a failure-aware observability framework for diagnosing wasted computation in multi-agent LLM traces.<n>We instantiate the framework in a three- agent question-answering system and evaluate it on 165 GAIA validation traces under identical execution caps.
  arXiv  Detail & Related papers  (2026-05-31T17:50:11Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• STAR: Failure-Aware Markovian Routing for Multi-Agent Spatiotemporal Reasoning [13.329339723676147]
  This paper presents STAR, a failureaware routing framework that externalizes interagent control as a state-conditioned policy over the current agent, task type, and typed execution status.<n>Because the traces on failure states are distinct, the router can respond differently to malformed outputs, missing dependencies, and tool-ground extract-- compute-deposit protocol and write intermediate results to a shared blackboard for downstream fusion.<n>Results prove that retaining unsuccessful traces during training enlarges the support of the routing policy on error states, enabling recovery transitions that success-only training cannot represent.
  arXiv  Detail & Related papers  (2026-05-11T06:34:49Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• ProbeLLM: Automating Principled Diagnosis of LLM Failures [89.44131968886184]
  We propose ProbeLLM, a benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes.<n>By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence.
  arXiv  Detail & Related papers  (2026-02-13T14:33:13Z)
• R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification [44.99719889905381]
  Reinforcement learning drives recent advances in reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation.<n>We propose R$3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification.<n>Experiments on agentic and reasoning tasks demonstrate 5% to 52% relative improvements over baselines while maintaining training stability.
  arXiv  Detail & Related papers  (2026-01-07T09:04:52Z)
• Uncertainty-aware Unsupervised Multi-Object Tracking [33.53331700312752]
  unsupervised multi-object trackers are inferior to learning reliable feature embeddings. Recent self-supervised techniques are adopted, whereas they failed to capture temporal relations. This paper argues that though the uncertainty problem is inevitable, it is possible to leverage the uncertainty itself to improve the learned consistency in turn.
  arXiv  Detail & Related papers  (2023-07-28T09:03:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.