Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

• URL: http://arxiv.org/abs/2608.11727v1
• Date: Wed, 12 Aug 2026 07:07:57 GMT
• Title: Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents
• Abstract summary: Harness-IF scores operational rules one at a time from execution evidence.<n>AP-Acc scores only rules labeled as opposing unprompted defaults.
• Score: 29.96375986740068
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success. We introduce Harness-IF, which scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads. To separate compliance from coincidence we introduce Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds and curated otherwise. Across 12 frontier models, accuracy spans 72.1-85.9% and AP-Acc 66.1-78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals. Aggregate scores therefore overstate compliance by a model-specific margin: prior control leaves the top build unchanged and exchanges three adjacent rank pairs. A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.

Related papers

• Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports [5.580052437190499]
  We reimplement the stated matching rule for only five of twelve inspected systems.<n>Ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings.<n>On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewers more than seven times in ten.
  arXiv  Detail & Related papers  (2026-09-01T04:45:44Z)
• QuoteBench: How Matched Scores Can Hide Command-Path Failures [30.480838412827907]
  Execution scores alone cannot distinguish command-generation errors from failures introduced after generation.<n> QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families.<n> Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state rather than treat a matched score as an intrinsic model property.
  arXiv  Detail & Related papers  (2026-08-13T17:57:20Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• ACCORD: Action-Conditioned Contextual Grounding for Language Agents [69.73525608707764]
  ACCORD (Action-Conditioned Contextual Grounding) is an agent framework for adaptive grounding.<n>We show that current agents often fail to do so. They act from assumed rather than observed specifics, overlook information they could have gathered, and fail to incorporate evidence that has already been returned.
  arXiv  Detail & Related papers  (2026-06-15T09:05:55Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• AgentAtlas: Beyond Outcome Leaderboards for LLM Agents [0.025718125188898048]
  AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol.<n>The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models.
  arXiv  Detail & Related papers  (2026-05-19T22:05:12Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• A Deterministic Agentic Workflow for HS Tariff Classification: Multi-Dimensional Rule Reasoning with Interpretable Decisions [16.99257993610188]
  The architecture combines offline knowledge-engineering of the Chinese tariff with an online six-stage pipeline.<n>A two-stage manual audit of 226 six-digit disagreements suggests that a non-trivial fraction of HSCodeComp ground-truth labels may deviate from general rules.
  arXiv  Detail & Related papers  (2026-05-14T14:04:46Z)
• Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance [0.0]
  This paper reports a single-system empirical case study of agent-driven auditing applied to AEGIS.<n>We report a seven-category post-hoc defect taxonomy with explicit coding rules, observed non-monotonic convergence, and audit-scope expansion.
  arXiv  Detail & Related papers  (2026-05-12T15:39:04Z)
• Do Agent Rules Shape or Distort? Guardrails Beat Guidance in Coding Agents [9.989306175511238]
  Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones.<n>Individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules.
  arXiv  Detail & Related papers  (2026-04-13T07:10:01Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.