SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories

• URL: http://arxiv.org/abs/2601.13713v1
• Date: Tue, 20 Jan 2026 08:10:56 GMT
• Title: SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
• Abstract summary: SWE-Tester is a novel pipeline for training open-source LLMs to generate issue reproduction tests.<n>First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories.<n>The fine-tuned models achieve absolute improvements of up to 10% in success rate and 21% in change coverage on SWT-Bench Verified.
• Score: 4.70019882353957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descriptions enhances developer productivity by simplifying root cause analysis, promotes test-driven development -- "test first, write code later", and can be used for improving the effectiveness of automated issue resolution systems like coding agents. Existing methods proposed for this task predominantly rely on closed-source LLMs, with limited exploration of open models. To address this, we propose SWE-Tester -- a novel pipeline for training open-source LLMs to generate issue reproduction tests. First, we curate a high-quality training dataset of 41K instances from 2.6K open-source GitHub repositories and use it to train LLMs of varying sizes and families. The fine-tuned models achieve absolute improvements of up to 10\% in success rate and 21\% in change coverage on SWT-Bench Verified. Further analysis shows consistent improvements with increased inference-time compute, more data, and larger models. These results highlight the effectiveness of our framework for advancing open-source LLMs in this domain.

Related papers

• Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation [1.4490522938516996]
  We focus on automated test case generation using Large Language Models (LLMs)<n>For the latter, we enable inspection of the source code by LLMs.<n>Our experimental results indicate that incorporating external context via the RAG pipeline has a generally positive impact on both test case generation and code inspection.
  arXiv  Detail & Related papers  (2026-04-16T17:41:49Z)
• TestDecision: Sequential Test Suite Generation via Greedy Optimization and Reinforcement Learning [17.157244731153554]
  We formalize test suite generation as a MDP and demonstrate that its objective exhibits monotone submodularity.<n>We propose TestDecision, which transforms LLMs into neural greedy experts.<n>TestDecision significantly outperforms existing advanced methods.
  arXiv  Detail & Related papers  (2026-04-02T09:13:52Z)
• Test-time Recursive Thinking: Self-Improvement without External Feedback [120.80790108733942]
  Test-time Recursive Thinking (TRT) is an iterative self-improvement framework.<n>Open-source models reach 100% accuracy on AIME-25/24, and on LiveCodeBench's most difficult problems, closed-source models improve by 10.4-14.8 percentage points without external feedback.
  arXiv  Detail & Related papers  (2026-02-03T04:37:37Z)
• An Experimental Study of Real-Life LLM-Proposed Performance Improvements [2.503024366864326]
  Large Language Models (LLMs) can generate code, but can they generate fast code?<n>We study this question using a dataset of 65 real-world tasks mined from open-source Java programs.
  arXiv  Detail & Related papers  (2025-10-17T10:06:52Z)
• Test-Time Learning for Large Language Models [33.11605667376906]
  We propose a Test-Time Learning (TTL) paradigm for Large Language Models (LLMs)<n>LLMs dynamically adapts to target domains using only unlabeled test data during testing.<n>We demonstrate through experiments that TLM improves performance by at least 20% compared to original LLMs on domain knowledge adaptation.
  arXiv  Detail & Related papers  (2025-05-27T02:18:59Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection [72.92366526004464]
  Retrieval-Augmented Generation (RAG) has proven effective in enabling Large Language Models (LLMs) to produce more accurate and reliable responses.<n>We propose a novel Self-Selection RAG framework, where the LLM is made to select from pairwise responses generated with internal parametric knowledge solely.
  arXiv  Detail & Related papers  (2025-02-10T04:29:36Z)
• Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation [12.503002900186997]
  Large Language Models (LLMs) have gained popularity for automated test case generation.<n>Because LLMs are trained on vast amounts of open-source code, they often generate test cases that do not adhere to best practices.<n>We propose Reinforcement Learning from Static Quality Metrics (RLSQM) to generate high-quality unit tests based on static analysis-based quality metrics.
  arXiv  Detail & Related papers  (2024-12-18T20:20:01Z)
• Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding [61.45448947483328]
  We introduce Lossless Acceleration via Speculative Decoding for LLM-based Recommender Systems (LASER)<n>LASER features a Customized Retrieval Pool to enhance retrieval efficiency and Relaxed Verification to improve the acceptance rate of draft tokens.<n>LASER achieves a 3-5x speedup on public datasets and saves about 67% of computational resources during the online A/B test.
  arXiv  Detail & Related papers  (2024-08-11T02:31:13Z)
• On the Evaluation of Large Language Models in Unit Test Generation [16.447000441006814]
  Unit testing is an essential activity in software development for verifying the correctness of software components. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation.
  arXiv  Detail & Related papers  (2024-06-26T08:57:03Z)
• DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning [56.887047551101574]
  We present DS-Agent, a novel framework that harnesses large language models (LLMs) agent and case-based reasoning (CBR) In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle. In the deployment stage, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm, significantly reducing the demand on foundational capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-02-27T12:26:07Z)
• Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation [12.503002900186997]
  Large Language Models (LLMs) have gained popularity for code generation, including the automated creation of test cases.<n>LLMs are often trained on vast amounts of publicly available code, which may include test cases that do not adhere to best practices.<n>We propose a novel technique called Reinforcement Learning from Static Quality Metrics (RLSQM)
  arXiv  Detail & Related papers  (2023-10-03T18:48:31Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.