Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol

• URL: http://arxiv.org/abs/2608.20729v1
• Date: Fri, 21 Aug 2026 04:21:28 GMT
• Title: Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol
• Abstract summary: Language-model agents can improve after failure or carry text across episodes without revising what counts as success.<n>We evaluate CMB-0.1 on twelve cross-domain cases and four arms.<n>No model trial satisfies all five conditions, but this zero does not establish general capability absence.
• Score: 3.202978695204522
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Language-model agents can improve after failure or carry text across episodes without revising what counts as success. We study the narrower attribution problem of criterion revision: when criterion K0 accepts an outcome violating a broader commitment B, what observations justify saying that the system formed and persistently used K1? We require five non-compensatory conditions: criterion-failure detection, a model-emitted proposal, new-episode transfer, intervention sensitivity on the claimed carrier, and preservation. We evaluate CMB-0.1 on twelve cross-domain cases and four arms: stateless inference, append-only history, model-generated but harness-committed state, and evaluator-written oracle state. Seven mechanism fixtures yield 84 deterministic scorer trials; four local quantized artifacts yield 96 calls and 192 model-case-arm trials. No model trial satisfies all five conditions, but this zero does not establish general capability absence. Eleven calls remain invalid after one retry; several commitments disclose the target distinction; the harness performs commits; deletion reuses a stateless call; and conflict changes multiple factors. Qwen2.5-7B answers every transfer and preservation item without revision state, exposing zero-state reconstruction. These failures make CMB-0.1 an instrument-calibration result rather than a model ranking. We derive a prospective, trace-anchored CMB-0.4 protocol requiring concealed transfer, explicit WRITE/NO-WRITE/ESCALATE actions, a separately logged policy-selected commit, matched interventions, repeated hidden items, and a frozen executable oracle. It is a successor design, not a completed confirmatory result. The paper contributes a measurement chain, an empirical diagnosis of its first implementation, and a more discriminating protocol for future tests of criterion revision.

Related papers

• From Experiments to Decisions: Reusing Evidence in Autonomous Coding Research [0.0]
  We reconstruct how evidence is reused in a 400-task NeuroGolf campaign.<n>A numerical counterexample exposes an overbroad exclusion.<n>Other episodes distinguish what failed, incomplete and revised programs justify doing next.
  arXiv  Detail & Related papers  (2026-09-10T03:48:31Z)
• Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores [31.709549159768727]
  We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
  arXiv  Detail & Related papers  (2026-08-31T16:43:55Z)
• Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence [0.0]
  Two instances of one model, in a two-agent handoff, co-fail on 90.0% of the missions on which either fails.<n>Substituting a different model reduces the association in six of six contrasts.
  arXiv  Detail & Related papers  (2026-08-13T07:25:05Z)
• Governed Persistent Memory: Source-Bound State Semantics and Fail-Closed Release for Long-Horizon Agents [3.202978695204522]
  Long-term agent memory is usually treated as select--store--retrieve, but retrieval does not decide whether contradictory, superseded, retracted, deleted, or stale records may support an outgoing claim.<n>We introduce Governed Persistent Memory (GPM), an auditable bitemporal state-transition model with source-bound admission, derived lifecycle state, current public barriers, and fail-closed structured release.
  arXiv  Detail & Related papers  (2026-08-12T18:00:42Z)
• Best Friends, Not Forever: Evaluating Long-Horizon Persona Collapse and Behavioral Drift in AI Companions [34.277389276831435]
  We study two observable long-horizon failures: 'persona collapse', the loss of a deployed role, boundaries, values, or style, and 'behavioral drift'<n>We introduce ANCHOR, a controlled synthetic audit that separately measures persona enactment and trajectory recall.<n>The study contains 2,008 conversations spanning 27 personas, nine interaction schedules, three generated memory settings, and four evaluated models.
  arXiv  Detail & Related papers  (2026-07-30T20:18:39Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing [35.76482964927589]
  We test the hypothesis that concept-specific memory traces become linear objects that can be extracted once and combined arithmetically.<n>We run that test on the authors' own reference implementation, at their reported best edit strength.<n>For unlearning-as-compliance: erasure certified today does not certify the artifact after its next edit.
  arXiv  Detail & Related papers  (2026-07-06T23:45:06Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents [11.233308795768465]
  VIGIL is an evaluation framework that makes terminal commitment independently measurable.<n>Under VIGIL's default protocol, agents observe only egocentric RGB, receive no action-success signals, and must end each episode with a semantic report checked deterministically against hidden world state.<n>This yields two separate scores: world-state completion (W) and benchmark success (B)
  arXiv  Detail & Related papers  (2026-05-09T07:24:07Z)
• Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning [32.32593439144886]
  Behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification.<n>Our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation.
  arXiv  Detail & Related papers  (2025-12-22T22:51:48Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.