DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences

• URL: http://arxiv.org/abs/2607.16024v1
• Date: Fri, 17 Jul 2026 14:57:59 GMT
• Title: DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences
• Abstract summary: This paper presentsDiffTestGen, a change-directed, LLM-based differential testing approach.<n>We show that DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%.<n>By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.
• Score: 12.692695524978332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As software evolves over time, it is important to ensure that any behavioral changes occur as intended by developers. A promising approach for this goal is to generate tests that expose behavioral differences between the old and new versions of a program. However, current approaches fail to trigger behavioral differences for many code changes. This paper presents~DiffTestGen, a novel change-directed, LLM-based differential testing approach specifically designed to expose behavioral differences introduced by a code change. The approach is enabled by two key contributions: First, DiffTestGen leverages static call graph analysis and project documentation to identify valid entry points for test generation and to guide the LLM toward reaching the changed code. Second, DiffTestGen iteratively improves our newly introduced union coverage metric, which combines coverage of modified code in the old and the new version, by providing targeted coverage feedback to the LLM. We evaluate DiffTestGen on two datasets comprising a total of 463 PRs. DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%. Compared with the baselines, DiffTestGen exposes 99 more PRs overall and increases code coverage by 12.5% and 15.6% percentage points, respectively. By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.

Related papers

• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation [4.061618191574875]
  Spec-Driven Test Generation instructs an agent to first reason about -- and explicitly document -- code pre-conditions, post-conditions, and undefined behaviors.<n>This intermediate semi-formal specification acts as a cognitive scaffold to guide subsequent test generation.
  arXiv  Detail & Related papers  (2026-08-17T22:36:12Z)
• Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study) [6.228248513679207]
  Studies show that correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled.<n>We show that the usefulness of coverage and mutation is highly context-dependent.<n>We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection.
  arXiv  Detail & Related papers  (2026-07-24T19:46:30Z)
• SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests [5.3138104073306005]
  Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining.<n>SemaDiff is a novel approach for identifying semantic-preserving commits through comparison of similar test execution on pre- and post-commit versions.<n>SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.
  arXiv  Detail & Related papers  (2026-07-14T12:23:22Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• Evaluating LLM-Based Test Generation Under Software Evolution [7.140756378584939]
  Large Language Models (LLMs) are increasingly used for automated unit test generation.<n>We present a large-scale empirical study of LLM-based test generation under program changes.
  arXiv  Detail & Related papers  (2026-03-24T17:14:18Z)
• Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning [54.95476453942411]
  Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates.<n>Recent self-play methods unify code and test generation in a single model.<n>Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests.
  arXiv  Detail & Related papers  (2026-03-16T17:58:13Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns [50.401907401444404]
  Large language models (LLMs) are crucial for preventing misuse and building trustworthy AI systems.<n>We propose RepreGuard, an efficient statistics-based detection method.<n> Experimental results show that RepreGuard outperforms all baselines with average 94.92% AUROC on both in-distribution (ID) and OOD scenarios.
  arXiv  Detail & Related papers  (2025-08-18T17:59:15Z)
• Testora: Using Natural Language Intent to Detect Behavioral Regressions [16.852583199679117]
  This paper presents Testora, the first automated approach that detects regressions by comparing the intentions of a code change against behavioral differences caused by the code change.<n>We find 19 regression bugs and 11 PRs that, despite having another intention, coincidentally fix a bug.<n>We envision our approach to be used before or shortly after a code change gets merged into a code base, providing a way to early on detect regressions that are not caught by traditional approaches.
  arXiv  Detail & Related papers  (2025-03-24T11:55:35Z)
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation [64.9230895853942]
  Domain generalization can be arbitrarily hard without exploiting target domain information. Test-time adaptive (TTA) methods are proposed to address this issue. In this work, we adopt Non-Parametric to perform the test-time Adaptation (AdaNPC)
  arXiv  Detail & Related papers  (2023-04-25T04:23:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.