Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail

• URL: http://arxiv.org/abs/2608.23651v1
• Date: Mon, 24 Aug 2026 10:19:17 GMT
• Title: Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
• Abstract summary: Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model.<n>The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent harnesses record a failed tool call and its error message in the transcript and ask the model to continue, on the assumption that the error is corrective information. We measure whether it is. Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model we tested (6 checkpoints, 135M-1.7B, 4 families) in two environments: simulated tool calling and MBPP program repair. Normalised by action length the effect is about -1.03 nats per action token, a factor of 2.8 in the odds of each token, and holds on 90%-100% of individual items, not only on average. Over a fixed candidate set the probability of repeating the failed call rises from 0.06 to 0.54, and greedy decoding reproduces it token for token on 19% of items after the failure versus 0% before. Counterfactuals pairing the same call with a failure message, a success message, or a neutral acknowledgement separate two effects: the failed call's surface form accounts for 83% of the damage, while the semantic contribution of marking it failed is small and inconsistent in sign across environments. The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work. Replacing the verbatim call with a runtime-generated description of the failure removes 76% of the inversion at no token cost, and making previously-failed strings unreachable at the decoder acts on the same term. Two plausible remedies do not: an explicit "do not repeat" instruction leaves the measured quantity where it was, and deleting the failed attempt to retry from a clean context, the standard prescription for context contamination, is the worst harness we measured for repetition, because it restores the context that produced the failure. The study runs end to end on a CPU; all artefacts are released.

Related papers

• Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return [7.146717861821737]
  Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one.<n>We isolate this post-failure decision with a benchmark of 1,024 items spanning 16 internal-system domains and eight tool-failure types.
  arXiv  Detail & Related papers  (2026-09-13T19:46:42Z)
• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• Invocation-Level Reliability of Tool-Using Agents [0.7874708385247353]
  We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context.<n>By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes.
  arXiv  Detail & Related papers  (2026-08-23T09:04:59Z)
• Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents [22.30468215041597]
  A tool-using agent is given the same task in a different language, does it still take the same steps?<n>We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages.
  arXiv  Detail & Related papers  (2026-08-11T16:18:34Z)
• Real-Time Detection and Repair of LLM Agent Failures [0.0]
  LLM agents fail mid-episode loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content.<n>We ask how much detection is achievable from observable step alone, using monitors costing microseconds per step and trained only on healthy runs.<n>System runs at 200 microseconds per step, three orders of magnitude below a judge call.
  arXiv  Detail & Related papers  (2026-08-03T16:34:46Z)
• Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets [13.85834524293015]
  We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with.<n>Models judge whether a candidate belongs far better than they author the set itself.
  arXiv  Detail & Related papers  (2026-08-02T05:00:44Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Code Correctness Signals in LLM Hidden States: Pre-Generation Probing and Repair Geometry [0.0]
  The correctness of the model's first-attempt code is linearly decodable from the prompt-final hidden state.<n>On 236 cleaned cases where the model attempts to repair a failed first attempt, the hidden state shift from the failing attempt to its repair carries a statistically detectable contrastive direction.
  arXiv  Detail & Related papers  (2026-06-12T15:06:15Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Demonstrating a long-coherence dual-rail erasure qubit using tunable transmons [59.63080344946083]
  We show that a "dual-rail qubit" consisting of a pair of resonantly coupled transmons can form a highly coherent erasure qubit. We demonstrate mid-circuit detection of erasure errors while introducing $ 0.1%$ dephasing error per check. This work establishes transmon-based dual-rail qubits as an attractive building block for hardware-efficient quantum error correction.
  arXiv  Detail & Related papers  (2023-07-17T18:00:01Z)
• SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition [116.31926128970585]
  We propose SoftCorrect with a soft error detection mechanism to avoid the limitations of both explicit and implicit error detection. Compared with implicit error detection with CTC loss, SoftCorrect provides explicit signal about which words are incorrect. Experiments on AISHELL-1 and Aidatatang datasets show that SoftCorrect achieves 26.1% and 9.4% CER reduction respectively.
  arXiv  Detail & Related papers  (2022-12-02T09:11:32Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.