A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

• URL: http://arxiv.org/abs/2608.18389v1
• Date: Tue, 18 Aug 2026 23:46:11 GMT
• Title: A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations
• Abstract summary: We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding code variations is rewritten into a semantically equivalent form.<n>We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs)<n>We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models.
• Score: 7.48361039542955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI code agents are increasingly deployed to resolve real software issues, yet their reliability under superficial code variations remains poorly understood. We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding codebase is rewritten into a semantically equivalent form. We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants. We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models (Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B) across instances drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agent is run multiple times on the unperturbed and perturbed variants, yielding paired resolve-rate estimates that isolate the perturbation effect from intrinsic stochasticity. We find small degradation in most configurations: up to 6.7 percentage points mean resolve-rate drop in the most affected configurations with statistically significant degradations in 6 of 16 configurations of model, scaffold, and dataset. Crucially, no single model ranking by robustness holds across scaffolds - Qwen is among the most robust under mini-SWE agent on SWE-bench Verified yet the most brittle under OpenCode - revealing a jagged robustness frontier. The simpler scaffold (mini-SWE agent) is more robust to perturbation. Our results demonstrate that even top frontier models are susceptible to semantics-preserving perturbations although the effect is not uniform, raising concerns about the deployment reliability of AI code agents in diverse real-world codebases.

Related papers

• A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption [33.31789388282026]
  Coding agents increase development velocity but also technical debt.<n>Prior work reports only average effects across adopters, hiding wide differences between teams.<n>We introduce RAMP, a four-level cumulative maturity model grounded in version-controlled artifacts that teams commit to configure AI tools.
  arXiv  Detail & Related papers  (2026-08-26T00:00:48Z)
• KAT-Coder-V2.5 Technical Report [61.907486544595336]
  KAT-Coder-V2.5 is a coding-focused agentic model trained to act autonomously inside real, executable repositories.<n>It reconstructs multilingual repositories into sandboxed environments with fail-to-pass and pass-to-pass verification at scale.<n>We further scale reinforcement learning with harness randomization, a reliability-hardened sandbox, an asymmetric actor--critic PPO with hindsight-augmented value estimation, and a harness-oriented reward framework.
  arXiv  Detail & Related papers  (2026-07-06T08:14:02Z)
• Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection [80.15317858033534]
  ForeAgent is an agentic forensics framework for AI-generated image detection with iterative self-evolution.<n>ForeAgent achieves state-of-the-art performance on the Chameleon benchmark, reaching 82.18% accuracy.<n>ForeAgent produces more consistent and causally grounded reasoning compared to GPT-5 and GPT-5-mini.
  arXiv  Detail & Related papers  (2026-06-25T02:59:33Z)
• Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution [4.960870628748691]
  Inefficient context management can lead to rapid context degradation and context poisoning.<n>We propose icat-agent, a decentralized, multi-agent scaffolding that replaces shared context with synchronous, event-based message passing.<n>icat-agent consistently outperforms prominent baselines across all difficulty levels.
  arXiv  Detail & Related papers  (2026-06-24T07:48:05Z)
• ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures [1.0742675209112622]
  We study failure modes as stemming from three mechanisms: underspecification, capability errors, and agent harness errors.<n>We propose ClayBuddy, a harness that molds to user preferences and can be modified by the model in-session to mitigate these errors.
  arXiv  Detail & Related papers  (2026-06-13T18:29:53Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• StaAgent: An Agentic Framework for Testing Static Analyzers [7.951459111292028]
  StaAgent is an agentic framework that harnesses the generative capabilities of Large Language Models (LLMs) to systematically evaluate static analyzer rules.<n>StaAgent helps uncover flaws in rule implementations by revealing inconsistent behaviors.<n>We evaluated StaAgent with five state-of-the-art LLMs across five widely used static analyzers.
  arXiv  Detail & Related papers  (2025-07-20T13:41:02Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.