Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

• URL: http://arxiv.org/abs/2605.29054v2
• Date: Wed, 03 Jun 2026 18:44:55 GMT
• Title: Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
• Abstract summary: Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
• Score: 56.25095230687242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Coding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but this progress has exposed a critical weakness: agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks while violating the semantic contracts users actually care about. This problem is especially acute in codebase conversion, where prior evaluation is largely outcome-driven and therefore unstable: two implementations can match on a shallow outcome, such as a single forward loss, while diverging in gradients, optimizer behavior, or short-horizon training dynamics. We introduce T2J-Bench, a benchmark for codebase conversion that reformulates conversion as transfer under a fixed equivalence contract. A fixed verifier then compares source and converted codebases through three ordered stages: Spec (interface admissibility), Numeric (forward outputs, losses, gradients, and objective-specific tensors), and Behavioral (short training dynamics under fixed seeds). Across 355 blind conversion attempts, the best system reaches only 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%; a 4.7x token-budget spread yields only a 2.2x pass-rate spread; and all systems overestimate success by 66.6--97.8 points relative to the fixed evaluator. This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.

Related papers

• Robustness of LLM-Generated SystemVerilog Assertions to Semantics-Preserving RTL Transformations [0.0]
  Large language models (LLMs) are increasingly being explored for automated SystemVerilog Assertion (SVA) generation.<n>Most evaluations report correctness on a single syntactic representation of an input.<n>Such point accuracy does not reveal whether a model's correct output is stable when the same RTL behavior is written differently.
  arXiv  Detail & Related papers  (2026-09-04T18:41:58Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes [40.20394793623452]
  When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome.<n>We propose Hierarchical Advantage-Weighted Behavior Cloning (HABC), which trains separate critic heads for viability and efficiency.<n>HABC raises success from supervised fine-tuning (SFT) baselines of 36%, 44%, and 12% to 92%, 88%, and 38%.
  arXiv  Detail & Related papers  (2026-06-15T17:57:14Z)
• CRANE: Constrained Reasoning Injection for Code Agents via Nullspace Editing [5.661334639541121]
  We present CRANE, a training-free parameter-editing method that treats the Thinking-Instruct delta as a pool of candidate reasoning edits for the Instruct backbone.<n>By merging paired Instruct and Thinking checkpoints, CRANE delivers strong gains over either individual model.
  arXiv  Detail & Related papers  (2026-05-13T20:09:35Z)
• StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning [16.3498616105159]
  We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision.<n>By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step.<n>Our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.
  arXiv  Detail & Related papers  (2026-05-12T10:36:56Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• A Benchmark for Evaluating Repository-Level Code Agents with Intermediate Reasoning on Feature Addition Task [11.218318079376365]
  RACE-bench is a reasoning-augmented benchmark for evaluating code agents on feature addition tasks.<n> RACE-bench contains 528 real-world feature addition instances from 12 open-source repositories.<n>We evaluate three representative repository-level code agents on RACE-bench.
  arXiv  Detail & Related papers  (2026-03-27T11:58:47Z)
• TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis [23.834704102474927]
  Code agents can resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why.<n>We introduce TRAJEVAL, a diagnostic framework that decomposes agent trajectories into three interpretable stages.<n>We validate that these diagnostics are predictive, achieving model-level Pass@1 prediction within 0.87-2.1% MAE.
  arXiv  Detail & Related papers  (2026-03-25T05:27:03Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• Scaling the Scaling Logic: Agentic Meta-Synthesis of Logic Reasoning [18.75349680577575]
  SSLogic is a framework for continuous family evolution with controllable difficulty.<n>Training on SSLogic-evolved data yields consistent gains over the seed baseline at matched steps.
  arXiv  Detail & Related papers  (2026-01-23T13:26:01Z)
• Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding [58.92526489742584]
  We propose provably lossless.<n> verification method that significantly boosts the expected number of accepted tokens.<n>We show that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks.
  arXiv  Detail & Related papers  (2026-01-09T11:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.