AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?

• URL: http://arxiv.org/abs/2607.03332v1
• Date: Fri, 03 Jul 2026 13:51:22 GMT
• Title: AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?
• Abstract summary: Coding agents often finish a session by returning one squashed patch that mixes feature implementation, bug fixes, partitionings, tests, and configuration edits.<n>We study retrospective commit-history reconstruction: given a completed squashed change, an agent groups its hunks into commits and materializes a replayable commit sequence.<n>We build AtomicCommitBench, containing 800 real consecutive-commit episodes from 10 Python projects.
• Score: 9.689734830501429
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Coding agents often finish a session by returning one squashed patch that mixes feature implementation, bug fixes, refactorings, tests, and configuration edits. While the final code may be correct, collapsing unrelated edits into one patch removes the history structure needed for review, selective revert, and later maintenance. We study retrospective commit-history reconstruction: given a completed squashed change, an agent groups its hunks into commits and materializes a replayable commit sequence. We formalize the task as hunk-to-commit partitioning with a replay requirement and build AtomicCommitBench, containing 800 real consecutive-commit episodes from 10 Python projects. Because multiple decompositions may be reasonable, we evaluate outputs using complementary metrics: PPAR for replay validity, ARI for reference-based grouping quality, and TCR for failure containment on scoreable modified-test episodes. Natural retrospective reconstruction proves substantially harder than replay checking or synthetic tangling. Although nearly all non-random methods achieve replay validity (PPAR >= 0.988), grouping quality ranges from 0.03 to 0.46 ARI. Matched synthetic composites are much easier than real same-author squashed diffs (+0.333 ARI). In our evaluation, the GPT-5.4 setup driven by Codex CLI (0.46 ARI) and the GLM-5 setup driven by Claude Code (0.43 ARI) outperform MiniMax (0.31) and Kimi (0.29). Qualitative analysis identifies same-file lumping and support-hunk drift as recurring failure modes. Dependency-Aware Commit Evidence (DACE) improves the lower-scoring setups by 0.05 to 0.08 ARI, indicating that dependency cues and hunk-role information help agents avoid locality-driven grouping errors. AtomicCommitBench enables evaluation of the commit histories produced by coding agents alongside the final code.

Related papers

• Beneath the Diff: Diagnosing and Mitigating Algorithmic Mode Collapse in Code-Level Autonomous Research Loops [11.99667127177886]
  We show that repeated metric-driven code editing leads to genuine improvements that generalize beyond the loop.<n>We propose a lightweight mitigation that combines category-coverage reweighting, persistent edit memory, and a validation gate.
  arXiv  Detail & Related papers  (2026-08-31T08:11:19Z)
• Temporal Validity on Real Software Histories: Eliminating Stale-Fact Errors in Code-Assistant Memory over GitHub Fixes [0.0]
  Retrieval-augmented generation (RAG) has no model of time.<n>RAG retrieves both the old and new value with near-identical similarity and cannot tell which is current, so it serves the superseded value.<n>MemStrata reaches 0.91 answer accuracy versus RAG's 0.57-0.59; and, the structural result, when forced to answer RAG serves the superseded value 36-38% of the time.
  arXiv  Detail & Related papers  (2026-08-21T02:48:36Z)
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework [49.16055123488827]
  coding agents powered by large language models can now generate code patches directly from bug reports.<n>No mechanism independently verifies whether it truly resolves the reported problem.<n>We propose RETRACE, a training-free post-generation verification framework.
  arXiv  Detail & Related papers  (2026-08-09T22:59:13Z)
• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
  Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
  arXiv  Detail & Related papers  (2026-07-22T07:30:07Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Why Git Is the Memory Solution for the Agentic Development Lifecycle [0.0]
  We argue memory should be git-bound, built into the repository's version control.<n>On this ledger we solve two problems separately, then combine them.<n>A router dispatches breadth to a git-anchored structural map, pointed lookups to confidence-gated episodes, and rationale to decision synthesis.
  arXiv  Detail & Related papers  (2026-07-15T22:06:59Z)
• Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework [0.3867363075280543]
  We present a closed-loop framework in which every accepted review comment is codified as a persistent behavioral rule.<n>The framework combines an accumulating rule set in a version-controlled instruction file, a self-review checklist, and automated validation.<n>We show that our framework achieves persistent cross-session learning without weight updates, operates on productions rather than synthetic benchmarks, and addresses an experiential dimension (behavioral consistency over time) that existing benchmarks do not measure.
  arXiv  Detail & Related papers  (2026-07-13T21:13:46Z)
• CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis [0.0]
  Decomposing compound sentences into atomic, verifiable claims is a prerequisite for reliable automated fact-checking.<n>Prior work has relied on token-overlap (Jaccard) metrics that systematically underestimate decomposition quality for paraphrastic claims.<n>We present Credence, a revised claim decomposition and evaluation framework addressing both shortcomings.
  arXiv  Detail & Related papers  (2026-06-18T05:48:37Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination [11.65571332626047]
  TeamTR is a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control.<n>We show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement.
  arXiv  Detail & Related papers  (2026-05-01T23:42:57Z)
• Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate [8.424102114588559]
  Automated Program Repair (APR) struggles with complex logic errors and silent failures.<n>Current LLM-based APR methods are mostly static, relying on source code and basic test outputs.<n>We propose TraceRepair, a multi-agent framework that leverages runtime facts as shared constraints for patch validation.
  arXiv  Detail & Related papers  (2026-04-03T02:23:25Z)
• APEX-EM: Non-Parametric Online Learning for Autonomous Agents via Structured Procedural-Episodic Experience Replay [7.370176470430802]
  LLM-based autonomous agents lack persistent procedural memory.<n>We present APEX-EM, a non-parametric online learning framework that accumulates, retrieves, and reuses structured procedural plans.
  arXiv  Detail & Related papers  (2026-03-31T00:24:56Z)
• Outcome-Conditioned Reasoning Distillation for Resolving Software Issues [49.16055123488827]
  We present an Outcome-Conditioned Reasoning Distillation(O-CRD) framework that uses resolved in-repository issues with verified patches as supervision.<n>Starting from a historical fix, the method reconstructs a stage-wise repair trace backward from the verified outcome.<n>On SWE-Bench Lite, this approach increases Pass@1 by 10.4% with GPT-4o, 8.6% with DeepSeek-V3, and 10.3% with GPT-5.
  arXiv  Detail & Related papers  (2026-01-30T18:25:39Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement [1.980982378865332]
  ARCS operates through a budgeted synthesize-execute-repair loop over a frozen model.<n>It retrieves relevant code context before generation, proposes candidates, executes them against tests, and repairs based on execution feedback.<n>On a LANL scientific corpus, it improves CodeBLEU by +0.115 over baseline RAG.
  arXiv  Detail & Related papers  (2025-04-29T05:15:52Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Detection Transformer with Stable Matching [48.963171068785435]
  We show that the most important design is to use and only use positional metrics to supervise classification scores of positive examples. Under the principle, we propose two simple yet effective modifications by integrating positional metrics to DETR's classification loss and matching cost. We achieve 50.4 and 51.5 AP on the COCO detection benchmark using ResNet-50 backbones under 12 epochs and 24 epochs training settings.
  arXiv  Detail & Related papers  (2023-04-10T17:55:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.