Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study

• URL: http://arxiv.org/abs/2503.15223v1
• Date: Wed, 19 Mar 2025 14:02:21 GMT
• Title: Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study
• Authors: You Wang, Michael Pradel, Zhongxin Liu,
• Abstract summary: Most popular benchmarks for automated issue solving are SWE-bench and its human-filtered subset SWE-bench Verified.<n>This paper presents an in-depth empirical study of the correctness of plausible patches generated by three state-of-the-art issue-solving tools evaluated on SWE-bench Verified.
• Score: 20.46588369793562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated issue solving aims to resolve real-world issues in software repositories. The most popular benchmarks for automated issue solving are SWE-bench and its human-filtered subset SWE-bench Verified. These benchmarks leverage testing to validate generated patches. However, because testing is rarely exhaustive, a patch may pass the tests but nevertheless fail to match the developers' expectations. Unfortunately, it is currently unclear to what extent evaluations performed with SWE-bench suffer from such plausible but incorrect patches. This paper presents an in-depth empirical study of the correctness of plausible patches generated by three state-of-the-art issue-solving tools evaluated on SWE-bench Verified. We extensively test and inspect generated patches, and compare them against human-written ground truth patches. The core of our methodology is a novel technique PatchDiff for differential patch testing, which automatically exposes behavioral discrepancies between two patches. Our findings reveal critical weaknesses in SWE-bench's patch validation mechanism, which causes 7.8% of all patches to count as correct while failing the developer-written test suite. Moreover, our novel automated technique reveals that even more (29.6%) plausible patches induce different behavior than the ground truth patches. These behavioral differences are often due to similar, but divergent implementations (46.8%) and due to generated patches that adapt more behavior than the ground truth patches (27.3%). Our manual inspection shows that 28.6% of behaviorally divergent patches are certainly incorrect. Combined, the different weaknesses lead to an inflation of reported resolution rates by 6.2 absolute percent points. Our findings are a call to arms for more robust and reliable evaluation of issue-solving tools. We envision our automated differential patch testing technique to be useful for this purpose.

Related papers

• All Patches Matter, More Patches Better: Enhance AI-Generated Image Detection via Panoptic Patch Learning [76.79222779026634]
  We establish two key principles for AIGI detection through systematic analysis. textbf(1) All Patches Matter: Unlike conventional image classification where discriminative features concentrate on object-centric regions, each patch in AIGIs inherently contains synthetic artifacts due to the uniform generation process. textbf (2) More Patches Better: Leveraging distributed artifacts across more patches improves detection by capturing complementary forensic evidence. textbfPanoptic textbfPatch textbfLearning (PPL) framework.
  arXiv  Detail & Related papers  (2025-04-02T06:32:09Z)
• Show Me Why It's Correct: Saving 1/3 of Debugging Time in Program Repair with Interactive Runtime Comparison [18.933377426587015]
  We propose an interactive approach called iFix to facilitate patch understanding and comparison.<n>iFix performs static analysis to identify runtime variables related to the buggy statement.<n>It captures runtime values during execution for each patch, allowing users to compare and contrast their runtime behavior.
  arXiv  Detail & Related papers  (2025-03-01T20:52:49Z)
• SoftPatch+: Fully Unsupervised Anomaly Classification and Segmentation [84.07909405887696]
  This paper is the first to consider fully unsupervised industrial anomaly detection (i.e., unsupervised AD with noisy data)<n>We propose memory-based unsupervised AD methods, SoftPatch and SoftPatch+, which efficiently denoise the data at the patch level.<n>Compared with existing methods, SoftPatch maintains a strong modeling ability of normal data and alleviates the overconfidence problem in coreset.<n> Comprehensive experiments conducted in diverse noise scenarios demonstrate that both SoftPatch and SoftPatch+ outperform the state-of-the-art AD methods on the MVTecAD, ViSA, and BTAD benchmarks.
  arXiv  Detail & Related papers  (2024-12-30T11:16:49Z)
• Patch-aware Batch Normalization for Improving Cross-domain Robustness [55.06956781674986]
  Cross-domain tasks present a challenge in which the model's performance will degrade when the training set and the test set follow different distributions. We propose a novel method called patch-aware batch normalization (PBN) By exploiting the differences between local patches of an image, our proposed PBN can effectively enhance the robustness of the model's parameters.
  arXiv  Detail & Related papers  (2023-04-06T03:25:42Z)
• PatchZero: Zero-Shot Automatic Patch Correctness Assessment [13.19425284402493]
  We propose toolname, the patch correctness assessment by adopting a large language model for code. toolname prioritizes labeled patches from existing APR tools that exhibit semantic similarity to those generated by new APR tools. Our experimental results showed that toolname can achieve an accuracy of 84.4% and an F1-score of 86.5% on average.
  arXiv  Detail & Related papers  (2023-03-01T03:12:11Z)
• Test-based Patch Clustering for Automatically-Generated Patches Assessment [21.051652050359852]
  Overfitting happens when a patch is run and the test suite does not reveal any error, but the patch actually does not fix the underlying bug or it introduces a new defect that is not covered by the test suite. Our work aims to minimize the number of plausible patches that programmers have to review, thereby reducing the time required to find a correct patch. We introduce a novel light-weight test-based patch clustering approach called xTestCluster, which clusters patches based on their dynamic behavior.
  arXiv  Detail & Related papers  (2022-07-22T13:39:27Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)
• Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detection [142.24869736769432]
  Adversarial patch attacks pose a serious threat to state-of-the-art object detectors. We propose Segment and Complete defense (SAC), a framework for defending object detectors against patch attacks. We show SAC can significantly reduce the targeted attack success rate of physical patch attacks.
  arXiv  Detail & Related papers  (2021-12-08T19:18:48Z)
• PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing [7.88628640954152]
  Vision Transformer (ViT) is known to be highly nonlinear like other classical neural networks and could be easily fooled by both natural and adversarial patch perturbations. This limitation could pose a threat to the deployment of ViT in the real industrial environment, especially in safety-critical scenarios. We propose PatchCensor, aiming to certify the patch robustness of ViT by applying exhaustive testing.
  arXiv  Detail & Related papers  (2021-11-19T23:45:23Z)
• Checking Patch Behaviour against Test Specification [4.723400023753107]
  We propose a hypothesis on how the link between the patch behaviour and failing test specifications can be drawn. We then propose BATS, an unsupervised learning-based system to predict patch correctness.
  arXiv  Detail & Related papers  (2021-07-28T11:39:06Z)
• (De)Randomized Smoothing for Certifiable Defense against Patch Attacks [136.79415677706612]
  We introduce a certifiable defense against patch attacks that guarantees for a given image and patch attack size. Our method is related to the broad class of randomized smoothing robustness schemes. Our results effectively establish a new state-of-the-art of certifiable defense against patch attacks on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2020-02-25T08:39:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.