SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

• URL: http://arxiv.org/abs/2607.13111v1
• Date: Tue, 14 Jul 2026 12:23:22 GMT
• Title: SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests
• Abstract summary: Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining.<n>SemaDiff is a novel approach for identifying semantic-preserving commits through comparison of similar test execution on pre- and post-commit versions.<n>SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.
• Score: 5.3138104073306005
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining. While existing approaches detect refactoring commits accurately, they cannot ensure that a commit is purely semantic-preserving, without any interleaving behaviour-changing modification. This limitation can impact several tasks, such as debugging, fault localisation, bug dataset construction, rollback analysis, and bug fixes backporting. To fill this gap, we propose SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis; comparison of similar test execution on pre- and post-commit versions. As code impacted by the refactoring is often hard to test and different accross both versions, we propose generating additional calling methods to that code, which serve as testing target. Given a commit, SemaDiff analyses the diff to identify modified code and extracts unchanged dependent code that calls it. It then generates an additional dependent class using a large language model to exercise the changed code in both versions, and automatically generates tests for the dependent code. This way, we obtain the same tests for the different code versions, enabling the behavioural-difference detection. The commit is classified as semantic-preserving only if all generated tests produce identical outcomes across the two versions. To evaluate SemaDiff, we construct and annotate manually a dataset of 183 commits, gathered from well-known open-source Java projects. The obtained results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.

Related papers

• DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences [12.692695524978332]
  This paper presentsDiffTestGen, a change-directed, LLM-based differential testing approach.<n>We show that DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%.<n>By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.
  arXiv  Detail & Related papers  (2026-07-17T14:57:59Z)
• What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents [14.988334585821939]
  Coding agents are ranked almost entirely by resolve rate.<n>A single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens.<n>We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides.
  arXiv  Detail & Related papers  (2026-07-07T12:09:46Z)
• IntentTester: Intent-Driven Multi-agent Framework for Cross-Library Test Migration [14.0813614333238]
  We present IntentTester, a framework for intent-driven test reuse.<n>We evaluate IntentTester on nine open-source projects across three domains (JSON, HTML, and Time) and two languages (Java and Python)
  arXiv  Detail & Related papers  (2026-06-24T08:59:01Z)
• Efficient Test-Time Inference via Deterministic Exploration of Truncated Decoding Trees [68.04613115686509]
  Self-consistency boosts inference-time performance by sampling multiple reasoning traces in parallel and voting.<n>We propose Distinct Leafion (DLE), a deterministic decoding method that treats truncated sampling as a pruned tree and systematically enumerates distinct leaves instead of sampling with replacement.<n>DLE explores higher-quality reasoning traces, yielding better performance on math, coding, and general reasoning tasks.
  arXiv  Detail & Related papers  (2026-04-22T12:42:03Z)
• Treating Run-time Execution History as a First-Class Citizen: Co-Versioning Run-time Behavior alongside Code [2.6397379133308214]
  This paper argues that this mismatch contributes to a blind spot in software evolution analysis and CI.<n>We propose textitBehavioral Co-Versioning, a paradigm that couples the Git history with a textitBehavioral Archive.<n>This enables semantic diffing, behavior-aware regression localization, and retrospective auditing by querying historical executions.
  arXiv  Detail & Related papers  (2026-04-18T09:36:12Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset [0.0]
  We present ReDef, a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects.<n>Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks.<n>This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior existing resources.
  arXiv  Detail & Related papers  (2025-09-11T07:07:11Z)
• Turning the Tide: Repository-based Code Reflection [52.13709676656648]
  We introduce LiveRepoReflection, a benchmark for evaluating code understanding and generation in multi-file repository contexts.<n>1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty.<n>We also create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources.
  arXiv  Detail & Related papers  (2025-07-14T02:36:27Z)
• A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools [2.0625936401496237]
  Syntax Tree (AST) diff tools were developed to overcome the limitations of line-based diff tools, which are used by the majority of developers. We propose a novel AST diff tool based on RefactoringMiner that resolves all aforementioned limitations. Our tool achieved a considerably higher precision and recall, especially for commits, with an execution time that is comparable with incompatible tools.
  arXiv  Detail & Related papers  (2024-03-09T15:32:41Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis [55.15995704119158]
  We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. We use D2A to generate a large labeled dataset to train models for vulnerability identification.
  arXiv  Detail & Related papers  (2021-02-16T07:46:53Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.