RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents

• URL: http://arxiv.org/abs/2609.04898v1
• Date: Fri, 04 Sep 2026 08:58:10 GMT
• Title: RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents
• Abstract summary: We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis.<n>Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction.
• Score: 0.09320657506524148
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, yet to our knowledge no existing harness isolates the design choices that determine agent success on this task. We present RefactorPlatform, an open-source evaluation harness that holds the environment fixed and varies each design axis explicitly: model backbone (via OpenRouter and GitHub Copilot CLI), execution regime (baseline, retrieval-augmented, and multi-agent), and prompt specificity. Each run executes in an isolated workspace with live terminal streaming, per-task logging of tokens, diffs, and transcripts, AST-based verification, and exportable telemetry for audit and reproduction. Demonstrating the platform on 100 multi-file RefactorBench tasks across four model families, we illustrate the analyses it supports: AST-aware chunking outperforms naive token-window chunking by 25-30% across prompt modes, whereas naive retrieval falls below the retrieval-free baseline; a lean retrieval-augmented single agent (86%) beats the sub-agent configuration we evaluated (66%) on matched tasks with no task passing under delegation that fails under retrieval; and retrieval's accuracy gains absorb its token overhead, leaving cost per successful refactoring unchanged. RefactorPlatform is open-sourced to make refactoring-agent evaluation reproducible and auditable.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring [3.294884078309524]
  REFINE is a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level candidates.<n>We evaluate REFINE on 450 Java files from 15 open-source systems.
  arXiv  Detail & Related papers  (2026-08-21T17:44:40Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents [5.604484678527336]
  CLOSER-Bench is a controlled evaluation protocol for budgeted cross-stage design closure.<n>It pairs spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, records every simulator, synthesis, STA, and place-and-route invocation.<n>It measures final quality, anytime progress, tool cost, and cross-stage recovery.
  arXiv  Detail & Related papers  (2026-07-18T04:28:47Z)
• AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows [1.6785141970297952]
  AEVAL (Agentic Evaluation) is a CI-integrated framework that replaces this practice with a deterministic, reproducible test pipeline for agentic skills.<n>Every skill change triggers a test event: the skill runs against a developer declared evaluation contract inside an automated executor.<n>A key ingredient is a structural separation between executor and grader, preventing a subtle but pervasive failure mode.
  arXiv  Detail & Related papers  (2026-07-16T21:33:05Z)
• RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations [51.43574078961796]
  We introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified.<n>RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access.<n>We propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving.
  arXiv  Detail & Related papers  (2026-05-25T06:26:43Z)
• ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment [0.9623578875486184]
  Digital libraries curate millions of research software artefacts yet lack scalable infrastructure for assessing whether those artefacts remain executable.<n>We present ReproScore, a two-tier framework that explicitly separates readiness (RRS) from outcome (ROS)<n>RRS comprises 26 sub-metrics across five categories; ROS provides execution-based probes when sandbox infrastructure is available; a community externalises weighting priorities as versioned YAML profiles.
  arXiv  Detail & Related papers  (2026-05-13T09:54:06Z)
• Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning [79.88942231770629]
  Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs)<n>We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR)<n>We synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement.
  arXiv  Detail & Related papers  (2026-04-08T00:53:29Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• RefAgent: A Multi-agent LLM-based Framework for Automatic Software Refactoring [8.038518812060897]
  RefAgent is a multi-agent LLM-based framework for end-to-end software.<n>It consists of specialized agents responsible for planning, executing, and iteratively refining testing.<n>It achieves a median unit test pass rate of 90%, reduces code smells by a median of 52.5%, and improves key quality attributes by a median of 8.6%.
  arXiv  Detail & Related papers  (2025-11-05T03:20:58Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code [7.156224931977546]
  We introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file tasks in popular open-source repositories.<n> Baselines reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions.<n>By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks.
  arXiv  Detail & Related papers  (2025-03-10T20:23:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.