Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback

• URL: http://arxiv.org/abs/2607.01360v1
• Date: Wed, 01 Jul 2026 18:20:27 GMT
• Title: Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback
• Abstract summary: PAIR-Bench is a progressive benchmark for evaluating large language models (LLMs)<n>It transforms an incorrect or incomplete program into a more correct one through feedback-guided refinement.<n>It measures whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires.
• Score: 13.976149104483449
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are typically evaluated on code generation and program repair using binary functional correctness: a generated program or patch either passes or fails a test suite. This protocol is simple but coarse, as it ignores partial progress, feedback use, regressions, and the refinement trajectory through which models often improve code. We introduce PAIR-Bench, a progressive and adaptive benchmark for evaluating code improvement: transforming an incorrect or incomplete program into a more correct one through feedback-guided refinement. PAIR-Bench uses progressive hinting, a structured feedback protocol with two controls. Failure-region control determines what the feedback targets by grouping hidden failing tests into failure scenarios, while hint-depth control determines how much repair-relevant information is revealed, from coarse symptoms to implementation-level guidance. This design enables PAIR-Bench to measure whether a model repairs targeted failures, generalizes beyond the hint, preserves already-correct behavior, and how much assistance it requires. By evaluating repair trajectories progressive metrics rather than only final pass/fail outcomes, PAIR-Bench provides a finer-grained assessment of LLM code-improvement capability.

Related papers

• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors [52.619793817715326]
  We present a multi-stage crowdsourced study evaluating skill-targeted, LLM-generated Python error messages.<n>We show that while LLM-rewritten messages significantly improved subjective evaluations, these perceived gains did not translate into statistically significant improvements in objective debug performance.<n>This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers.
  arXiv  Detail & Related papers  (2026-08-21T09:14:28Z)
• Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework [0.3867363075280543]
  We present a closed-loop framework in which every accepted review comment is codified as a persistent behavioral rule.<n>The framework combines an accumulating rule set in a version-controlled instruction file, a self-review checklist, and automated validation.<n>We show that our framework achieves persistent cross-session learning without weight updates, operates on productions rather than synthetic benchmarks, and addresses an experiential dimension (behavioral consistency over time) that existing benchmarks do not measure.
  arXiv  Detail & Related papers  (2026-07-13T21:13:46Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents [4.908588441456849]
  ProcCtrlBench is a benchmark for execution-process evaluation in LLM coding agents.<n>It organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories.<n>It evaluates agent trajectories through standardized process evidence rather than final outcomes alone.
  arXiv  Detail & Related papers  (2026-05-18T08:34:48Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Imperfect Visual Verification for Code Edition : A Case Study on TikZ [3.6213861298681826]
  We use TikZ as a focused case study that isolates the core difficulties of the problem.<n>We define visual code customization as an iterative editing problem with an imperfect oracle.<n>Our findings show that even imperfect verifiers can determine with moderate accuracy whether visual instructions are applied to code, achieving F1-scores up to 0.815.
  arXiv  Detail & Related papers  (2026-04-09T12:21:52Z)
• Beyond Fixed Tests: Repository-Level Issue Resolution as Coevolution of Code and Behavioral Constraints [17.818522356206977]
  Most large language model (LLM)-based repair systems treat behavioral constraints as fixed during repair.<n>We propose Agent-CoEvo, a coevolutionary multi-agent framework in which candidate code patches and test patches are jointly explored and refined.<n>Agent-CoEvo consistently outperforms state-of-the-art agent-based and agentless baselines in both repair success and test reproduction quality.
  arXiv  Detail & Related papers  (2026-04-06T10:26:46Z)
• TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code [11.207330722400764]
  We present TraceCoder, a framework that emulates the observe-analyze-repair process of human experts.<n>The framework first instruments the code with diagnostic probes to capture fine-grained runtime traces.<n>It then conducts causal analysis on these traces to accurately identify the root cause of the failure.
  arXiv  Detail & Related papers  (2026-02-06T16:59:48Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset [0.0]
  We present ReDef, a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects.<n>Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks.<n>This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior existing resources.
  arXiv  Detail & Related papers  (2025-09-11T07:07:11Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Teaching Your Models to Understand Code via Focal Preference Alignment [70.71693365502212]
  In existing approaches, a set of n candidate solutions is evaluated based on test case success rates.<n>Because this approach aligns entire failing code blocks rather than pinpointing specific errors, it lacks the granularity necessary to capture meaningful error-correction relationships.<n>We propose Target-DPO, a new preference alignment framework that mimics human iterative debug to refine Code LLMs.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• FixEval: Execution-based Evaluation of Program Fixes for Programming Problems [23.987104440395576]
  We introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately.
  arXiv  Detail & Related papers  (2022-06-15T20:18:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.