Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories

• URL: http://arxiv.org/abs/2606.07889v1
• Date: Fri, 05 Jun 2026 22:52:16 GMT
• Title: Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories
• Abstract summary: LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway.<n>We build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs.<n>We evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway. We call this pattern strained coherence: a safety-relevant failure mode in which an agent has information that should change its behavior, states that information, and still acts against it. The pattern overlaps with verbalized reward hacking, where an agent names a tension between a task proxy and the underlying goal yet optimizes the proxy anyway. We give an operational definition, build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs, and evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone. Flagged trajectories fail 94% of the time versus 46% for unflagged trajectories (47-point gap, Fisher's exact p = 0.003; 46 points after excluding three prompt-embedded examples, p = 0.006). At matched selectivity, the detector reaches 94% precision versus 88% for a lexical discourse-marker baseline; the 10-trajectory intersection of the two methods has a 100% failure rate (Clopper-Pearson 95% CI [69%, 100%]). We replicate on Gemma4-31B with 43 trajectories: the overall signal is directionally consistent but not significant (20-point gap, p = 0.31), with attenuation driven largely by 13 trajectories with zero think content, where the detector has no substrate to analyze. In the high-verbosity Gemma tertile, the gap is +30 points; in the mid- and high-verbosity Qwen tertiles, it is +40 points each. The first flag appears at a median of 83-84% of elapsed trajectory time across both models, and the binary flag survives paraphrases that soften explicit conflict markers (8/8 trajectories). Unlike univariate predictors, the detector emits interpretable span-level output -- quoted acknowledgment, quoted action, and typed conflict -- showing what the agent saw and ignored.

Related papers

• DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents [0.0]
  An operator needs three facts: where the attack entered, which steps it corrupted, and whether apparent poison was resisted.<n>We present DriftNet, a dual-head trajectory Transformer that reads a logged tool-call trajectory and answers all three questions in one forward pass.<n>To our knowledge it is the first supervised detector to produce this joint output.
  arXiv  Detail & Related papers  (2026-09-09T22:52:23Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• Reading Calibrated Uncertainty from Language Model Trajectories [46.663987199083245]
  Methods that probe the model's internal activations feed raw hidden states into opaque snapshots and leave implicit the layer-wise trajectory by which a representation is formed.<n>We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer updates, and feed them to a sparse linear probe.<n>The probe outperforms MSP under selective abstention, with baseline scaling with up to 21 AURC points.
  arXiv  Detail & Related papers  (2026-05-19T19:24:29Z)
• ContractBench: Can LLM Agents Preserve Observation Contracts? [9.057486468322933]
  We show that observation contract compliance is an emergent, regression-prone capability.<n>We introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two failure modes.<n>We report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B; (iii) non-monotonic scaling across the GPT-5 family: agentic post-training can erode
  arXiv  Detail & Related papers  (2026-05-17T06:37:04Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations [55.251494694783894]
  Disaster Operational Response Agent benchmark (DORA) is the first agentic benchmark for end-to-end disaster response.<n>Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis.<n>DORA establishes a rigorous testbed for operationally reliable disaster-response agents.
  arXiv  Detail & Related papers  (2026-05-12T06:57:41Z)
• Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection [0.0]
  Multi-turn prompt injection follows a known attack path.<n>We show this attack path leaves an activation-level signature in the model's residual stream.
  arXiv  Detail & Related papers  (2026-04-30T17:16:33Z)
• Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines [34.37967130277969]
  We propose a two-agent evasion framework operating in a semantic perturbation space.<n>Evasion rates are 19.95 to 40.34% on modern large language model (LLM) based systems.<n>A pattern-informed defense reduces the evasion rate by up to 65.18%.
  arXiv  Detail & Related papers  (2026-04-26T00:58:36Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers [34.31192184496381]
  Dense retrievers and rerankers are central to retrieval-augmented generation (RAG) pipelines.<n>We evaluate how much factual competence these components inherit or lose from large language models (LLMs) they are based on.<n>For every embedding model, cosine-similarity scores between queries and correct completions are significantly higher than those for incorrect ones.
  arXiv  Detail & Related papers  (2025-08-28T04:13:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.