Reproduction Test Generation for Java SWE Issues

• URL: http://arxiv.org/abs/2605.04320v2
• Date: Thu, 07 May 2026 19:11:05 GMT
• Title: Reproduction Test Generation for Java SWE Issues
• Abstract summary: This paper introduces both a benchmark and a solution for Java repository-level reproduction test generation.<n>The benchmark, TDD-Bench-Java, is the first to model this problem and comprises 250 instances sourced from popular open-source repositories.<n>The solution, e-Otter++ for Java, adapts a state-of-the-art reproduction test generator for Python to yield high performance on Java.
• Score: 7.216607755908824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execution-based feedback for diagnosis and validation during software development. Unfortunately, they are usually missing. Therefore, recent work has introduced both benchmarks and a thriving literature on solutions for reproduction test generation from issues. However, that work has focused on Python and neglected other languages such as Java, which is important for enterprise software. This paper introduces both a benchmark and a solution for Java repository-level reproduction test generation. The benchmark, TDD-Bench-Java, is the first to model this problem and comprises 250 instances sourced from popular open-source repositories. The solution, e-Otter++ for Java, adapts a state-of-the-art reproduction test generator for Python to yield high performance on Java. To evaluate in an industry setting, besides empirical results with TDD-Bench-Java, this paper also presents results with a contamination-free proprietary dataset. Overall, we hope that this paper contributes to bringing better diagnosis and validation to Java software development.

Related papers

• Resolving Java Code Repository Issues with iSWE Agent [3.927958209415049]
  iSWE Agent is an automated issue resolver with an emphasis on Java.<n>It consists of two sub-agents, one for localization and the other for editing.<n>It achieves state-of-the-art issue resolution rates across the Java splits of both Multi-SWE-bench and SWE-PolyBench.
  arXiv  Detail & Related papers  (2026-03-11T22:43:55Z)
• CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming [58.48683464644606]
  We introduce CPRet, a retrieval-oriented benchmark suite for competitive programming.<n>Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation.<n>We develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem-code alignment, and CPRetriever-Prob, fine-tuned for identifying problem-level similarity.
  arXiv  Detail & Related papers  (2025-05-19T10:07:51Z)
• EnvBench: A Benchmark for Automated Environment Setup [76.02998475135824]
  Large Language Models have enabled researchers to focus on practical repository-level tasks in software engineering domain.<n>Existing studies on environment setup introduce innovative agentic strategies, but their evaluation is often based on small datasets.<n>To address this gap, we introduce a comprehensive environment setup benchmark EnvBench.
  arXiv  Detail & Related papers  (2025-03-18T17:19:12Z)
• TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved? [11.762669773233474]
  Test-driven development (TDD) is the practice of writing tests first and coding later.<n>This paper introduces TDD-Bench Verified, a high-quality benchmark suite of 449 issues mined from real-world GitHub code repositories.
  arXiv  Detail & Related papers  (2024-12-03T22:38:05Z)
• Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs [21.06722050714324]
  This paper focuses on automation of test oracles for clients of widely used Java libraries, e.g., java.lang and java.util packages.<n>We use large language models as an enabling technology to embody our insight into a framework for test oracle automation.
  arXiv  Detail & Related papers  (2024-11-04T04:24:25Z)
• JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models [22.95865189208591]
  We propose JavaBench, a project-level Java benchmark that exercises OOP features. It comprises four Java projects with 389 methods in 106 Java classes. It is attested by 282 undergraduate students, reaching a 90.93/100 average score.
  arXiv  Detail & Related papers  (2024-06-10T06:43:25Z)
• BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies [8.746971239693066]
  For the first time, Python outperformed Java in Stack Overflow developer survey. This is in stark contrast with the abundance of testing and debug tools for Java. In this project, we create a benchmark database and tool that contain 493 real bugs from 17 real-world Python programs.
  arXiv  Detail & Related papers  (2024-01-27T19:07:34Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• AVATAR: A Parallel Corpus for Java-Python Program Translation [77.86173793901139]
  Program translation refers to migrating source code from one language to another. We present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python.
  arXiv  Detail & Related papers  (2021-08-26T05:44:20Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.