Does SWE-Bench-Verified Test Agent Ability or Model Memory?

• URL: http://arxiv.org/abs/2512.10218v1
• Date: Thu, 11 Dec 2025 02:11:06 GMT
• Title: Does SWE-Bench-Verified Test Agent Ability or Model Memory?
• Authors: Thanosan Prathifkumar, Noble Saji Mathews, Meiyappan Nagappan,
• Abstract summary: SWE-Bench-Verified is a dataset comprising 500 issues.<n>This benchmark may overlap with model training data.<n>We test two Claude models that frequently appear in top-performing agents submitted to the benchmark.
• Score: 2.937612609787308
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: SWE-Bench-Verified, a dataset comprising 500 issues, serves as a de facto benchmark for evaluating various large language models (LLMs) on their ability to resolve GitHub issues. But this benchmark may overlap with model training data. If that is true, scores may reflect training recall, not issue-solving skill. To study this, we test two Claude models that frequently appear in top-performing agents submitted to the benchmark. We ask them to find relevant files using only issue text, and then issue text plus file paths. We then run the same setup on BeetleBox and SWE-rebench. Despite both benchmarks involving popular open-source Python projects, models performed 3 times better on SWE-Bench-Verified. They were also 6 times better at finding edited files, without any additional context about the projects themselves. This gap suggests the models may have seen many SWE-Bench-Verified tasks during training. As a result, scores on this benchmark may not reflect an agent's ability to handle real software issues, yet it continues to be used in ways that can misrepresent progress and lead to choices that favour agents that use certain models over strong agent design. Our setup tests the localization step with minimal context to the extent that the task should be logically impossible to solve. Our results show the risk of relying on older popular benchmarks and support the shift toward newer datasets built with contamination in mind.

Related papers

• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason [1.6249398255272318]
  We present empirical evidence that performance gains on SWE-Bench-Verified may be partially driven by memorization rather than genuine problem-solving.<n>We show that state-of-the-art models achieve up to 76% accuracy in identifying buggy file paths using only issue descriptions, without access to repository structure.<n>These findings raise concerns about the validity of existing results and underscore the need for more robust, contamination-resistant benchmarks.
  arXiv  Detail & Related papers  (2025-06-14T00:25:26Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools [54.63478102768333]
  Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions.<n>We propose a novel class of model-internal confidence estimators (MICE) to better assess confidence when calling tools.
  arXiv  Detail & Related papers  (2025-04-28T18:06:38Z)
• Automated Benchmark Generation for Repository-Level Coding Tasks [7.305342793164905]
  SetUpAgent is a fully automated system capable of historically accurate dependency setup, test execution, and result parsing.<n>We generate two new datasets: (i) SWEE-Bench an extended version of SWE-Bench encompassing hundreds of repositories, and (ii) SWA-Bench a benchmark focusing on applications rather than libraries.
  arXiv  Detail & Related papers  (2025-03-10T17:42:49Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• PyBench: Evaluating LLM Agent on various real-world coding tasks [13.347173063163138]
  PyBench is a benchmark covering five main categories of real-world tasks, covering more than 10 types of files. Our evaluations indicate that current open-source LLMs are struggling with these tasks. Our fine-tuned 8B size model: textbfPyLlama3 achieves an exciting performance on PyBench.
  arXiv  Detail & Related papers  (2024-07-23T15:23:14Z)
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark [93.57775429120488]
  We release LiveBench, the first benchmark that contains frequently-updated questions from recent information sources.<n>We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size.<n>Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time.
  arXiv  Detail & Related papers  (2024-06-27T16:47:42Z)
• CommitBench: A Benchmark for Commit Message Generation [22.03783968903916]
  We show that existing datasets exhibit various problems, such as the quality of the commit selection. We compile a new large-scale dataset, CommitBench, adopting best practices for dataset creation. We use CommitBench to compare existing models and show that other approaches are outperformed by a Transformer model pretrained on source code.
  arXiv  Detail & Related papers  (2024-03-08T09:56:45Z)
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? [80.52201658231895]
  SWE-bench is an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. We show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues.
  arXiv  Detail & Related papers  (2023-10-10T16:47:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.