Do Agent Rules Shape or Distort? Guardrails Beat Guidance in Coding Agents

• URL: http://arxiv.org/abs/2604.11088v1
• Date: Mon, 13 Apr 2026 07:10:01 GMT
• Title: Do Agent Rules Shape or Distort? Guardrails Beat Guidance in Coding Agents
• Abstract summary: Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones.<n>Individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules.
• Score: 9.989306175511238
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Developers increasingly guide AI coding agents through natural language instruction files (e.g., CLAUDE.md, .cursorrules), yet no controlled study has measured whether these rules actually improve agent performance or which properties make a rule beneficial. We scrape 679 such files (25,532 rules) from GitHub and conduct the first large-scale empirical evaluation, running over 5,000 agent runs with a state-of-the-art coding agent on SWE-bench Verified. Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones -- suggesting rules work through context priming rather than specific instruction. Negative constraints ("do not refactor unrelated code") are the only individually beneficial rule type, while positive directives ("follow code style") actively hurt -- a pattern we analyze through the lens of potential-based reward shaping (PBRS). Moreover, individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules. These findings expose a hidden reliability risk -- well-intentioned rules routinely degrade agent performance -- and provide a clear principle for safe agent configuration: constrain what agents must not do, rather than prescribing what they should.

Related papers

• HazardAuditor: From Executable Threats to Safer Computer-Use Agents [50.06351661912873]
  Existing guard models target static prompts and responses.<n>Existing safety platforms produce evaluation verdicts rather than the normalized supervision a guard model needs to learn.<n>We introduce HazardAuditor, an execution-grounded framework that closes both gaps.
  arXiv  Detail & Related papers  (2026-09-14T07:09:33Z)
• Grounded Normative Rule Generation with Structured Search [47.28595196084396]
  We formalize the problem as Grounded Normative Rule Synthesis (GNRS)<n>We introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG)<n>We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharterBench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses.
  arXiv  Detail & Related papers  (2026-08-23T05:43:42Z)
• Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents [29.96375986740068]
  Harness-IF scores operational rules one at a time from execution evidence.<n>AP-Acc scores only rules labeled as opposing unprompted defaults.
  arXiv  Detail & Related papers  (2026-08-12T07:07:57Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• AutoSpec: Safety Rule Evolution for LLM Agents via Inductive Logic Programming [21.12573593471532]
  Existing safety approaches face a fundamental tradeoff: hand-crafted rules are interpretable but brittle.<n>We present AutoSpec, a framework that automatically evolves deployed expert-guided safety rules from user safe/unsafe annotations.<n>We evaluate AutoSpec on 291 execution traces spanning code execution and embodied agent domains.
  arXiv  Detail & Related papers  (2026-06-23T07:31:03Z)
• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement [120.52289344734415]
  We propose an automated framework for stealthy prompt injection tailored to agent skills.<n>The framework forms a closed loop with three agents: an Attack Agent that synthesizes injection skills under explicit stealth constraints, a Code Agent that executes tasks using the injected skills and an Evaluate Agent that logs action traces.<n>Our method consistently achieves high attack success rates under realistic settings.
  arXiv  Detail & Related papers  (2026-02-15T16:09:48Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Deriving Character Logic from Storyline as Codified Decision Trees [67.01182739162142]
  Role-playing (RP) agents rely on behavioral profiles to act consistently across diverse narrative contexts.<n>We propose Codified Decision Trees (CDT), a data-driven framework that induces an executable and interpretable decision structure from large-scale narrative data.
  arXiv  Detail & Related papers  (2026-01-15T05:12:43Z)
• Are Your Agents Upward Deceivers? [73.1073084327614]
  Large Language Model (LLM)-based agents are increasingly used as autonomous subordinates that carry out tasks for users.<n>This raises the question of whether they may also engage in deception, similar to how individuals in human organizations lie to superiors to create a good image or avoid punishment.<n>We observe and define agentic upward deception, a phenomenon in which an agent facing environmental constraints conceals its failure and performs actions that were not requested without reporting.
  arXiv  Detail & Related papers  (2025-12-04T14:47:05Z)
• Executable Governance for AI: Translating Policies into Rules Using LLMs [1.388831902854619]
  Policy-to-Tests (P2T) is a framework that converts natural policy documents into normalized, machine-readable rules.<n>To test the framework beyond a single policy, we apply it across general frameworks, sector guidance, and enterprise standards.<n>These AI-generated rules closely match strong human baselines on span-level and rule-level metrics, with robust inter-annotator agreement on the gold set.
  arXiv  Detail & Related papers  (2025-12-04T03:11:54Z)
• RulePilot: An LLM-Powered Agent for Security Rule Generation [15.638534363355342]
  Rule-based detection often identifies malicious logs based on the predefined grammar logic.<n>In this paper, we propose RulePilot, which mimics human expertise via LLM-based agent.<n>Using RulePilot, the security analysts do not need to write down the rules following the grammar, instead, they can just provide the annotations.
  arXiv  Detail & Related papers  (2025-11-15T13:59:16Z)
• RuleAgent: Discovering Rules for Recommendation Denoising with Autonomous Language Agents [36.31706728494194]
  RuleAgent mimics real-world data experts to autonomously discover rules for recommendation denoising.<n>LossEraser-an unlearning strategy streamlines training without compromising denoising performance.
  arXiv  Detail & Related papers  (2025-03-30T09:19:03Z)
• AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents [8.290987399121343]
  We propose AgentSpec, a lightweight language for specifying and enforcing runtime constraints on LLM agents.<n>With AgentSpec, users define structured rules that incorporate triggers, predicates, and enforcement mechanisms.<n>We implement AgentSpec across multiple domains, including code execution, embodied agents, and autonomous driving.
  arXiv  Detail & Related papers  (2025-03-24T13:31:48Z)
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents [50.81206098588923]
  RedCode is a benchmark for risky code execution and generation. RedCode-Exec provides challenging prompts that could lead to risky code execution. RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions.
  arXiv  Detail & Related papers  (2024-11-12T13:30:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.