Testora: Using Natural Language Intent to Detect Behavioral Regressions

• URL: http://arxiv.org/abs/2503.18597v3
• Date: Thu, 06 Nov 2025 16:22:17 GMT
• Title: Testora: Using Natural Language Intent to Detect Behavioral Regressions
• Abstract summary: This paper presents Testora, the first automated approach that detects regressions by comparing the intentions of a code change against behavioral differences caused by the code change.<n>We find 19 regression bugs and 11 PRs that, despite having another intention, coincidentally fix a bug.<n>We envision our approach to be used before or shortly after a code change gets merged into a code base, providing a way to early on detect regressions that are not caught by traditional approaches.
• Score: 16.852583199679117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As software is evolving, code changes can introduce regression bugs or affect the behavior in other unintended ways. Traditional regression test generation is impractical for detecting unintended behavioral changes, because it reports all behavioral differences as potential regressions. However, most code changes are intended to change the behavior in some way, e.g., to fix a bug or to add a new feature. This paper presents Testora, the first automated approach that detects regressions by comparing the intentions of a code change against behavioral differences caused by the code change. Given a pull request (PR), Testora queries an LLM to generate tests that exercise the modified code, compares the behavior of the original and modified code, and classifies any behavioral differences as intended or unintended. For the classification, we present an LLM-based technique that leverages the natural language information associated with the PR, such as the title, description, and commit messages -- effectively using the natural language intent to detect behavioral regressions. Applying Testora to PRs of complex and popular Python projects, we find 19 regression bugs and 11 PRs that, despite having another intention, coincidentally fix a bug. Out of 13 regressions reported to the developers, 11 have been confirmed and 9 have already been fixed. The costs of using Testora are acceptable for real-world deployment, with 12.3 minutes to check a PR and LLM costs of only $0.003 per PR. We envision our approach to be used before or shortly after a code change gets merged into a code base, providing a way to early on detect regressions that are not caught by traditional approaches.

Related papers

• Alteron: A Tool for Behavioral Regression Testing Across NLP Classifier Versions [0.3823356975862005]
  evaluating evolving Natural Language Processing (NLP) models is important for ensuring reliable behavior across updates.<n>Standard benchmark metrics do not fully capture how model behavior changes across versions.<n>We present Alteron, a tool for detecting behavioral regressions across NLP model versions with metamorphic testing.
  arXiv  Detail & Related papers  (2026-07-31T15:48:31Z)
• DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences [12.692695524978332]
  This paper presentsDiffTestGen, a change-directed, LLM-based differential testing approach.<n>We show that DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%.<n>By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.
  arXiv  Detail & Related papers  (2026-07-17T14:57:59Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents [99.67681154381803]
  Test-time Rule Acquisition and Compiled Enforcement (TRACE) is a skill-layer pipeline for coding-agent runtimes.<n>Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections.
  arXiv  Detail & Related papers  (2026-06-11T10:43:40Z)
• RippleGUItester: Change-Aware Exploratory Testing [22.40767870807728]
  We present RippleGUItester, a change-driven testing system that treats a code change as the epicenter of a ripple effect.<n>We evaluate our approach on hundreds of real-world code changes across four widely used software systems: Firefox, Zettlr, JabRef, and Godot.<n>In total, we identify 26 previously unknown bugs that still exist in the latest versions of the evaluated systems.
  arXiv  Detail & Related papers  (2026-03-03T15:56:49Z)
• Change And Cover: Last-Mile, Pull Request-Based Regression Test Augmentation [20.31612139450269]
  Testing pull requests (PRs) is critical to maintaining software quality.<n>Some PR-modified lines remain untested, leaving a "last-mile" regression test gap.<n>We present ChaCo, an LLM-based test augmentation technique that addresses this gap.
  arXiv  Detail & Related papers  (2026-01-16T02:08:16Z)
• ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration [68.89572566071575]
  ETAgent is a training framework for calibrating agent's tool-use behavior.<n>It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors.
  arXiv  Detail & Related papers  (2026-01-11T11:05:26Z)
• How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code? [56.42119949944239]
  We investigate whether semantically equivalent code transformation rules might be leveraged to evade MI detection.<n>We find that model accuracy drops by only 1.5% in the worst case for each rule.<n>Our results expose a critical loophole in license compliance enforcement for training large language models for code.
  arXiv  Detail & Related papers  (2025-12-17T14:12:54Z)
• Probing Pre-trained Language Models on Code Changes: Insights from ReDef, a High-Confidence Just-in-Time Defect Prediction Dataset [0.0]
  We present ReDef, a high-confidence benchmark of function-level modifications curated from 22 large-scale C/C++ projects.<n>Defective cases are anchored by revert commits, while clean cases are validated through post-hoc history checks.<n>This pipeline yields 3,164 defective and 10,268 clean modifications, offering substantially more reliable labels than prior existing resources.
  arXiv  Detail & Related papers  (2025-09-11T07:07:11Z)
• ReCatcher: Towards LLMs Regression Testing for Code Generation [11.185300073739098]
  ReCatcher is a regression testing framework for Python code generation.<n>We apply ReCatcher to assess regressions across three update scenarios, fine-tuning, merging, and model release.<n>Merging with general-purpose models like Llama2 leads to regressions in correctness by up to 18%.
  arXiv  Detail & Related papers  (2025-07-25T15:45:55Z)
• Evaluating the Effectiveness of Small Language Models in Detecting Refactoring Bugs [0.6133301815445301]
  This study evaluates the effectiveness of Small Language Models (SLMs) in detecting two types of bugs in Java and Python.<n>The study covers 16 types and employs zero-shot prompting on consumer-grade hardware to evaluate the models' ability to reason about correctness without explicit prior training.<n>The proprietary o3-mini-high model achieved the highest detection rate, identifying 84.3% of Type I bugs.<n>The open-source Phi-4 14B performed comparably well, demonstrating strong effectiveness across both bug types.
  arXiv  Detail & Related papers  (2025-02-25T18:52:28Z)
• Can LLM Generate Regression Tests for Software Commits? [15.653758694625854]
  Large Language Models (LLMs) have shown tremendous promise in automated software engineering.<n>We implement Cleverest, a feedback-directed zero-shot LLM-based regression test generation technique.<n>For programs using more human-readable file formats, like XML or JavaScript, Cleverest performed very well.
  arXiv  Detail & Related papers  (2025-01-19T15:46:26Z)
• Language Modeling with Editable External Knowledge [90.7714362827356]
  This paper introduces ERASE, which improves model behavior when new documents are acquired. It incrementally deletes or rewriting other entries in the knowledge base each time a document is added. It improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute.
  arXiv  Detail & Related papers  (2024-06-17T17:59:35Z)
• Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing [19.812562421377706]
  In NLP, incremental processors produce output in instalments, based on incoming prefixes of the linguistic input. Some tokens trigger revisions, causing edits to the output hypothesis, but little is known about why models revise when they revise. We investigate the appropriateness of regressions and skips in human reading eye-tracking data as signals to inform revision policies in incremental sequence labelling.
  arXiv  Detail & Related papers  (2023-10-27T16:08:15Z)
• Zero-Shot Detection of Machine-Generated Codes [83.0342513054389]
  This work proposes a training-free approach for the detection of LLMs-generated codes. We find that existing training-based or zero-shot text detectors are ineffective in detecting code. Our method exhibits robustness against revision attacks and generalizes well to Java codes.
  arXiv  Detail & Related papers  (2023-10-08T10:08:21Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• What learning algorithm is in-context learning? Investigations with linear models [87.91612418166464]
  We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly. We show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression. Preliminary evidence that in-context learners share algorithmic features with these predictors.
  arXiv  Detail & Related papers  (2022-11-28T18:59:51Z)
• Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness [8.606215760860362]
  We turn the patch correctness assessment into a Question Answering problem. We consider as inputs the bug reports as well as the natural language description of the generated patches. Experiments show that Quatrain can achieve an AUC of 0.886 on predicting patch correctness.
  arXiv  Detail & Related papers  (2022-08-08T13:32:58Z)
• AdaptivePaste: Code Adaptation through Learning Semantics-aware Variable Usage Representations [22.52158623717161]
  We introduce AdaptivePaste, a learning-based approach to source code adaptation. We evaluate AdaptivePaste on a dataset of code snippets in Python. Results suggest that our model can learn to adapt source code with 79.8% accuracy.
  arXiv  Detail & Related papers  (2022-05-23T03:49:48Z)
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates [68.09049111171862]
  This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. We formulate the regression-free model updates into a constrained optimization problem. We empirically analyze how model ensemble reduces regression.
  arXiv  Detail & Related papers  (2021-05-07T03:33:00Z)
• RepPoints V2: Verification Meets Regression for Object Detection [65.120827759348]
  We introduce verification tasks into the localization prediction of RepPoints. RepPoints v2 provides consistent improvements of about 2.0 mAP over the original RepPoints. We show that the proposed approach can more generally elevate other object detection frameworks as well as applications such as instance segmentation.
  arXiv  Detail & Related papers  (2020-07-16T17:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.