Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks

• URL: http://arxiv.org/abs/2604.05100v1
• Date: Mon, 06 Apr 2026 18:59:42 GMT
• Title: Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks
• Abstract summary: Instructed code editing accounts for roughly 19% of real-world coding assistant interactions.<n>From a survey of over 150 code-related benchmarks, we find that only two, CanItEdit and EDIT-Bench, target instructed code editing.
• Score: 2.5424331328233203
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Instructed code editing, where an LLM modifies existing code based on a natural language instruction, accounts for roughly 19% of real-world coding assistant interactions. Yet very few benchmarks directly evaluate this capability. From a survey of over 150 code-related benchmarks, we find that only two, CanItEdit and EDIT-Bench, target instructed code editing with human-authored instructions and test-based evaluation. We audit both by comparing their programming languages, edit intents, and application domains against distributions observed in the wild (Copilot Arena, AIDev, GitHub Octoverse), and by measuring test counts, statement coverage, and test scope across all 213 problems. Both benchmarks concentrate over 90\% of evaluation on Python while TypeScript, GitHub's most-used language, is absent. Backend and frontend development, which together constitute 46% of real-world editing activity, are largely missing, and documentation, testing, and maintenance edits (31.4% of human PRs) have zero representation. Both benchmarks have modest test counts (CanItEdit median 13, EDIT-Bench median 4), though CanItEdit compensates with near-complete whole-file coverage and fail-before/pass-after validation. 59\% of EDIT-Bench's low-coverage suites would not detect modifications outside the edit region. EDIT-Bench has 15 problems that are not solved by any of 40 LLMs and 11 of these problems trace failures to poor benchmark artifacts rather than model limitations. Further, 29% of EDIT-Bench problems and 6% of CanItEdit problems share a codebase with at least one other problem within the benchmark. In summary, these benchmarks measure a narrower construct than deployment decisions require. We therefore propose six empirically grounded desiderata and release all audit artifacts so the community can build instructed code-editing benchmarks whose scores reliably reflect real-world editing capability.

Related papers

• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• ProgramBench: Can Language Models Rebuild Programs From Scratch? [59.40748183470308]
  ProgramBench measures the ability of software engineering agents to develop software holisitically.<n>End-to-end behavioral tests are generated via agent-driven fuzzing.<n>Models favor monolithic, single-file implementations that diverge sharply from human-written code.
  arXiv  Detail & Related papers  (2026-05-05T09:17:02Z)
• SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing? [25.593993482047342]
  We propose SAFEdit, a multi-agent framework for instructed code editing.<n>A Planner Agent produces an explicit, visibility-aware edit plan, an Editor Agent applies minimal, literal code modifications, and a Verifier Agent executes real test runs.<n>When tests fail, SAFEdit uses a Failure Abstraction Layer to transform raw test logs into structured diagnostic feedback.
  arXiv  Detail & Related papers  (2026-04-28T15:04:46Z)
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits [72.23150343093447]
  We introduce EDIT-Bench, a benchmark for evaluatingstructed code editing capabilities grounded in real-world usage.<n>EDIT-Bench comprises of 545 problems, multiple natural and programming languages, and a diverse set of real-world use cases.<n>We find that model performance varies across different categories of user instructions.
  arXiv  Detail & Related papers  (2025-11-06T16:05:28Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• A Benchmark for Localizing Code and Non-Code Issues in Software Projects [26.511673758202267]
  We introduce MULocBench, a dataset of 1,100 issues from 46 popular GitHub Python projects.<n>Compared with existing benchmarks, MULocBench offers greater diversity in issue types, root causes, location scopes, and file types.<n>Using this benchmark, we assess the performance of state-of-the-art localization methods and five LLM-based prompting strategies.
  arXiv  Detail & Related papers  (2025-09-26T06:05:20Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• UTFix: Change Aware Unit Test Repairing using LLM [24.12850207529614]
  We present UTFix, a novel approach for repairing unit tests when their corresponding focal methods undergo changes.<n>Our approach leverages language models to repair unit tests by providing contextual information such as static code slices, dynamic code slices, and failure messages.<n>To the best of our knowledge, this is the first comprehensive study focused on unit test in evolving Python projects.
  arXiv  Detail & Related papers  (2025-03-19T06:10:03Z)
• Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications [4.751608548909266]
  FineEdit is a specialized editing model explicitly trained for context-aware text modifications.<n>FineEdit outperforms state-of-the-art models on single-turn edits, up to 30% over Llama-3.2-3B, and exceeding Mistral-7B-OpenOrca performance by over 40% on direct editing tasks.
  arXiv  Detail & Related papers  (2025-02-19T01:41:44Z)
• The Mirage of Model Editing: Revisiting Evaluation in the Wild [70.17413507444704]
  We introduce QAEdit, a new benchmark aligned with widely used question answering (QA) datasets, and WILD, a task-agnostic evaluation framework.<n>Our single editing experiments show that current editing methods perform substantially worse than previously reported.
  arXiv  Detail & Related papers  (2025-02-16T15:57:55Z)
• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases.<n>The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions [11.327913840111378]
  We introduce Defects4J-NL2Fix, a dataset of 283 Java programs from the popular Defects4J dataset augmented with high-level descriptions of bug fixes. We empirically evaluate the performance of several state-of-the-art LLMs for the this task.
  arXiv  Detail & Related papers  (2023-04-07T18:58:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.