When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation

• URL: http://arxiv.org/abs/2605.25981v2
• Date: Tue, 26 May 2026 13:50:43 GMT
• Title: When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation
• Abstract summary: We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
• Score: 9.055086193088083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models from seven architecture families: meaning-bearing perturbations (e.g., paraphrase, synonym) alter final answers more often than presentation perturbations (e.g., formatting, reordering) of comparable severity. Across 68 cells spanning GSM8K, MATH, and HotpotQA (1,530 originals and $\sim$11,150 variants), the inconsistency gap averages +19.69 pp after severity matching (paired $t=9.58$, $p<0.0001$), with 64/68 cells positive. The gap survives four severity-proxy audits and remains significant when excluding qwen models (+11.10 pp, $p<0.0001$). Several stress tests fail honestly: cluster-bootstrap significance disappears under stricter assumptions, tractability contrasts do not replicate, cross-architecture generator swaps break per-cell rankings, and a second LLM judge yields only moderate agreement ($κ=0.50$). We then validate the headline effect on a fully held-out 11th model (qwen2.5-14B-Instruct; 1,800 trajectories) and re-test a pre-registered capability$\times$tractability partition, observing a small but positive held-out effect (3/4 cells positive; pooled Welch $t=3.81$, $p=9.6\times10^{-4}$). Using held-out trajectories, we probe four trace-level mechanism signals. Two prior mechanism claims fail to replicate and are explicitly retracted. Two new probes instead support a \emph{stealth-divergence} picture: semantic perturbations often preserve the first action but induce divergence in intermediate reasoning from later steps onward, accompanied by slightly deeper trajectories. We position this as a measurement contribution with held-out replication and a partial trace-level account of how semantic perturbations propagate through agent reasoning. Code, perturbation corpus, raw trajectories, and analysis scripts are released anonymously for review.

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs [52.149036302760386]
  Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs.<n>Recent work proposes detection using linear probes that recover the secret from internal activations.<n>We show that this defense can be systematically evaded, but that detectability can be recovered through a targeted data-level intervention.
  arXiv  Detail & Related papers  (2026-06-08T12:27:11Z)
• Strained Coherence: A Pre-Failure Signal in Coding Agent Execution Trajectories [0.0]
  LLM-based coding agents sometimes acknowledge a problem in their own reasoning and then proceed anyway.<n>We build a Claude Sonnet 4.6 judge that reads full trajectories and flags spans where the pattern occurs.<n>We evaluate it on 44 Terminal-bench-2 trajectories using a Qwen3.5-35B-A3B backbone.
  arXiv  Detail & Related papers  (2026-06-05T22:52:16Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence [22.315546054051143]
  Bias audits of large language models now operate within governance frameworks such as the EU AI Act.<n>We introduce textscBiAxisAudit, a protocol that reports each bias score together with a reliability estimate on two axes.
  arXiv  Detail & Related papers  (2026-05-09T16:26:49Z)
• Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations [0.0]
  We find anti-concentration is observed in residual-stream difference-of-means vectors across five architecture families.<n>Results suggest transformers may rotate semantic content into spectrally quiet regions during contextualized processing.
  arXiv  Detail & Related papers  (2026-05-02T21:20:55Z)
• When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors [66.18091962164219]
  Existing metrics fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences.<n>We propose two complementary metrics to isolate non-mandatory behavioral patterns: textbfResponse Pattern Similarity (RPS) for verbal alignment and textbfAction Graph Similarity (AGS) for tool-use habits modeled as directed graphs.
  arXiv  Detail & Related papers  (2026-04-23T03:48:56Z)
• One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness [12.183451602438753]
  We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses.<n>We show that base models show no systematic collapse under identical constraints, with effects that are small, noisy, and bidirectional.
  arXiv  Detail & Related papers  (2026-04-14T17:40:01Z)
• Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements [66.94250413799232]
  We study mean estimation of a random vector $X$ in a distributed parameter-server-worker setup.<n>Key challenges are adversarial measurements and asynchrony.<n>Results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers.
  arXiv  Detail & Related papers  (2026-04-07T11:45:55Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Semantic Gravity Wells: Why Negative Constraints Backfire [0.0]
  Negative constraints (instructions of the form "do not use word X") represent a fundamental test of instruction-following capability in large language models.<n>This paper presents the first comprehensive mechanistic investigation of negative instruction failure.
  arXiv  Detail & Related papers  (2026-01-12T23:30:18Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Latency and Ordering Effects in Online Decisions [0.0]
  Online decision systems operate under delayed feedback and order-sensitive dynamics.<n>We package heterogeneous latency, noncommutativity, and implementation-gap effects into a single lower-bound statement.
  arXiv  Detail & Related papers  (2025-11-17T07:08:05Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)
• Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference [44.644520116360106]
  We estimate the global average treatment effect (GATE), the difference between average outcomes having exposed all units at all times to treatment or to control.<n>We propose a clustered switchback design, where units are grouped into clusters and time steps are grouped into blocks.<n>We show that for graphs that admit good clustering, a truncated Horvitz-Thompson estimator achieves a $tilde O(1/NT)$ mean squared error (MSE)<n>Our results simultaneously generalize the results from citethu2022switchback,ugander2013graph and citetleung2022rate
  arXiv  Detail & Related papers  (2023-12-25T01:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.