Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

• URL: http://arxiv.org/abs/2607.27250v1
• Date: Tue, 28 Jul 2026 11:07:53 GMT
• Title: Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
• Abstract summary: Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory.<n>We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.

Related papers

• Agent ATO: Visualizing Agent Interaction Timelines from Logs [1.5639957559514246]
  Agent ATO is a tool for visualizing AI coding agent interaction timelines from console logs.<n>In addition to an all-interaction timeline, Agent ATO provides filtered timelines that emphasize file discovery, file reading, file editing, and execution.
  arXiv  Detail & Related papers  (2026-09-08T06:22:44Z)
• Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents [2.9512821350868754]
  We present AgentEval, a black-box testing framework that discovers and stresses stateful boundaries.<n>Instead of prompting blindly, AgentEval uses this graph's structure to enumerate specific guards and prerequisites as test targets.<n>We benchmark AgentEval against a privileged, white-box auditor with access to the agent's underlying source code, which AgentEval never sees.
  arXiv  Detail & Related papers  (2026-07-08T00:11:41Z)
• AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation [0.18567307896771823]
  AgentLens is a production-assessed benchmark for interactive code agents.<n>It pairs formal verification with LLM-written trajectory reviews and side-by-side comparisons.<n>We use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline.
  arXiv  Detail & Related papers  (2026-07-07T11:27:43Z)
• ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures [1.0742675209112622]
  We study failure modes as stemming from three mechanisms: underspecification, capability errors, and agent harness errors.<n>We propose ClayBuddy, a harness that molds to user preferences and can be modified by the model in-session to mitigate these errors.
  arXiv  Detail & Related papers  (2026-06-13T18:29:53Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• How to Interpret Agent Behavior [56.59836196946289]
  We introduce ACT*ONOMY, a taxonomy for describing and analyzing agent behavior at runtime.<n>By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently.
  arXiv  Detail & Related papers  (2026-05-13T14:52:40Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications [0.0]
  Test-Driven AI Agent Definition (TDAD) is a methodology that treats agent prompts as compiled artifacts.<n>We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement.
  arXiv  Detail & Related papers  (2026-03-09T18:04:54Z)
• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement [120.52289344734415]
  We propose an automated framework for stealthy prompt injection tailored to agent skills.<n>The framework forms a closed loop with three agents: an Attack Agent that synthesizes injection skills under explicit stealth constraints, a Code Agent that executes tasks using the injected skills and an Evaluate Agent that logs action traces.<n>Our method consistently achieves high attack success rates under realistic settings.
  arXiv  Detail & Related papers  (2026-02-15T16:09:48Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• Explaining Reinforcement Learning Policies through Counterfactual Trajectories [147.7246109100945]
  A human developer must validate that an RL agent will perform well at test-time. Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
  arXiv  Detail & Related papers  (2022-01-29T00:52:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.