A Dataset of Reproducible Flaky-Test Failures

• URL: http://arxiv.org/abs/2605.21677v1
• Date: Wed, 20 May 2026 19:36:51 GMT
• Title: A Dataset of Reproducible Flaky-Test Failures
• Abstract summary: Flaky tests pass and fail non-deterministically when run on the same version of code.<n>We present ReproFlake, a dataset of 1115 reproducible flaky tests across four flaky test categories.
• Score: 6.23537633626453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Flaky tests pass and fail non-deterministically when run on the same version of code. Although many techniques have been proposed to detect, debug, and repair flaky tests, reproducing their failures remains a major challenge due to their inherent nondeterminism. Many flaky test datasets exist to help researchers study them, but these datasets are often composed of disjoint sets of flaky tests, where each dataset provides unique information, such as flaky tests of different categories, failure logs of flaky tests, or flaky tests reported by developers vs. flaky tests found by automated tools. In this work, we aim to create a reproducible dataset of flaky tests, curated from both developer issue reports and a popular dataset of flaky tests. Compared to prior flaky test datasets, our dataset is the first to provide (1) a reproducible environment to compile flaky tests, (2) scripts to reproduce failures, (3) scripts to automatically apply flaky test fixes and ensure that the tests are no longer flaky, and (4) execution logs of flaky test passing and failing. We present ReproFlake, a dataset of 1115 reproducible flaky tests across four flaky test categories. We create guidelines to help others contribute to this reproducible dataset, and demonstrate how to use our dataset to understand challenges in reproducing flaky test failures (e.g., challenges researchers may face when using prior flaky test datasets), the characteristics (e.g., location of the fix and its correlation with the flaky test category), and difficulties researchers may face in using our dataset to collect additional information (e.g., code coverage) about flaky tests. Our findings show that error information helps identify flaky test categories and guide repairs, that unresolved compilation failures highlight challenges in building legacy projects, and knowing typical fix locations can help prioritize repair efforts.

Related papers

• Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data [73.19814469893088]
  Digital twins and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems.<n>This paper addresses whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible.<n>Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing.
  arXiv  Detail & Related papers  (2026-08-28T07:05:38Z)
• Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA [45.467566253448666]
  Flaky tests yield different results when executed multiple times for the same version of the source code.<n>A variety of factors can cause test flakiness.<n> approaches to fix flaky tests are typically tailored to address specific causes.
  arXiv  Detail & Related papers  (2026-02-03T14:03:59Z)
• Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures [6.824747267214373]
  Flaky tests produce inconsistent outcomes without code changes.<n>Developers spend 1.28% of their time repairing flaky tests at a monthly cost of $2,250.<n>We show that flaky tests often exist in clusters, with co-occurring failures that share the same root causes, which we call systemic flakiness.
  arXiv  Detail & Related papers  (2025-04-23T14:51:23Z)
• Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANA [47.29324864511411]
  Flaky tests fail seemingly at random without changes to the code. We study characteristics of tests and the test environment that potentially impact test flakiness.
  arXiv  Detail & Related papers  (2024-09-16T07:52:09Z)
• Taming Timeout Flakiness: An Empirical Study of SAP HANA [47.29324864511411]
  Flaky tests negatively affect regression testing because they result in test failures that are not necessarily caused by code changes. Test timeouts are one contributing factor to such flaky test failures. Test flakiness rate ranges from 49% to 70%, depending on the number of repeated test executions.
  arXiv  Detail & Related papers  (2024-02-07T20:01:41Z)
• 230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers [9.45325012281881]
  Flaky tests are tests that can non-deterministically pass or fail, even in the absence of code changes. How to quickly determine if a test failed due to flakiness, or if it detected a bug?
  arXiv  Detail & Related papers  (2024-01-28T22:36:30Z)
• Do Automatic Test Generation Tools Generate Flaky Tests? [12.813573907094074]
  The prevalence and nature of flaky tests produced by test generation tools remain largely unknown. We generate tests using EvoSuite (Java) and Pynguin (Python) and execute each test 200 times. Our results show that flakiness is at least as common in generated tests as in developer-written tests.
  arXiv  Detail & Related papers  (2023-10-08T16:44:27Z)
• FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair [0.5749787074942512]
  Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test. In this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. One key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category.
  arXiv  Detail & Related papers  (2023-06-21T19:34:16Z)
• FlaPy: Mining Flaky Python Tests at Scale [14.609208863749831]
  FlaPy is a framework for researchers to mine flaky tests in a given or automatically sampled set of Python projects by rerunning their test suites. FlaPy isolates the test executions using containerization and fresh execution environments to simulate real-world CI conditions. FlaPy supports parallelizing the test executions using SLURM, making it feasible to scan thousands of projects for test flakiness.
  arXiv  Detail & Related papers  (2023-05-08T15:48:57Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.