Practical Pipeline-Aware Regression Test Optimization for Continuous Integration

• URL: http://arxiv.org/abs/2501.11550v1
• Date: Mon, 20 Jan 2025 15:39:16 GMT
• Title: Practical Pipeline-Aware Regression Test Optimization for Continuous Integration
• Authors: Daniel Schwendner, Maximilian Jungwirth, Martin Gruber, Martin Knoche, Daniel Merget, Gordon Fraser,
• Abstract summary: Continuous Integration (CI) is commonly applied to ensure consistent code quality. Developers commonly split test executions across multiple pipelines, running small and fast tests in pre-submit stages while executing long-running and flaky tests in post-submit pipelines. We developed a lightweight and pipeline-aware regression test optimization approach that employs Reinforcement Learning models trained on language-agnostic features.
• Score: 9.079940595000087
• License:
• Abstract: Massive, multi-language, monolithic repositories form the backbone of many modern, complex software systems. To ensure consistent code quality while still allowing fast development cycles, Continuous Integration (CI) is commonly applied. However, operating CI at such scale not only leads to a single point of failure for many developers, but also requires computational resources that may reach feasibility limits and cause long feedback latencies. To address these issues, developers commonly split test executions across multiple pipelines, running small and fast tests in pre-submit stages while executing long-running and flaky tests in post-submit pipelines. Given the long runtimes of many pipelines and the substantial proportion of passing test executions (98% in our pre-submit pipelines), there not only a need but also potential for further improvements by prioritizing and selecting tests. However, many previously proposed regression optimization techniques are unfit for an industrial context, because they (1) rely on complex and difficult-to-obtain features like per-test code coverage that are not feasible in large, multi-language environments, (2) do not automatically adapt to rapidly changing systems where new tests are continuously added or modified, and (3) are not designed to distinguish the different objectives of pre- and post-submit pipelines: While pre-submit testing should prioritize failing tests, post-submit pipelines should prioritize tests that indicate non-flaky changes by transitioning from pass to fail outcomes or vice versa. To overcome these issues, we developed a lightweight and pipeline-aware regression test optimization approach that employs Reinforcement Learning models trained on language-agnostic features. We evaluated our approach on a large industry dataset collected over a span of 20 weeks of CI test executions. When predicting...

Related papers

• Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation [67.18144414660681]
  We propose a Fast-Slow Test-Time Adaptation (FSTTA) approach for online Vision-and-Language Navigation (VLN) Our method obtains impressive performance gains on four popular benchmarks.
  arXiv  Detail & Related papers  (2023-11-22T07:47:39Z)
• FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs [92.47146416628965]
  FuzzyFlow is a fault localization and test case extraction framework designed to test program optimizations. We leverage dataflow program representations to capture a fully reproducible system state and area-of-effect for optimizations. To reduce testing time, we design an algorithm for minimizing test inputs, trading off memory for recomputation.
  arXiv  Detail & Related papers  (2023-06-28T13:00:17Z)
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation [64.9230895853942]
  Domain generalization can be arbitrarily hard without exploiting target domain information. Test-time adaptive (TTA) methods are proposed to address this issue. In this work, we adopt Non-Parametric to perform the test-time Adaptation (AdaNPC)
  arXiv  Detail & Related papers  (2023-04-25T04:23:13Z)
• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond [52.656743602538825]
  Fine-tuning pre-trained code models incurs a large computational cost. We conduct an experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We propose Telly to efficiently fine-tune pre-trained code models via layer freezing.
  arXiv  Detail & Related papers  (2023-04-11T13:34:13Z)
• Robust Continual Test-time Adaptation: Instance-aware BN and Prediction-balanced Memory [58.72445309519892]
  We present a new test-time adaptation scheme that is robust against non-i.i.d. test data streams. Our novelty is mainly two-fold: (a) Instance-Aware Batch Normalization (IABN) that corrects normalization for out-of-distribution samples, and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d. data stream from non-i.i.d. stream in a class-balanced manner.
  arXiv  Detail & Related papers  (2022-08-10T03:05:46Z)
• DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing [6.767885381740952]
  This work introduces DeepOrder, a deep learning-based model that works on the basis of regression machine learning. DeepOrder ranks test cases based on the historical record of test executions from any number of previous test cycles. We experimentally show that deep neural networks, as a simple regression model, can be efficiently used for test case prioritization in continuous integration testing.
  arXiv  Detail & Related papers  (2021-10-14T15:10:38Z)
• Reinforcement Learning for Test Case Prioritization [0.24366811507669126]
  This paper extends recent studies on applying Reinforcement Learning to optimize testing strategies. We test its ability to adapt to new environments, by testing it on novel data extracted from a financial institution. We also studied the impact of using Decision Tree (DT) Approximator as a model for memory representation.
  arXiv  Detail & Related papers  (2020-12-18T11:08:20Z)
• Neural Network Embeddings for Test Case Prioritization [0.24366811507669126]
  We have developed a new tool called Neural Network Embeeding for Test Case Prioritization (NNE- TCP) NNE- TCP analyses which files were modified when there was a test status transition and learns relationships between these files and tests by mapping them into multidimensional vectors. We show for the first time that the connection between modified files and tests is relevant and competitive relative to other traditional methods.
  arXiv  Detail & Related papers  (2020-12-18T10:33:28Z)
• MLCask: Efficient Management of Component Evolution in Collaborative Data Analytics Pipelines [29.999324319722508]
  We address two main challenges that arise during the deployment of machine learning pipelines, and address them with the design of versioning for an end-to-end analytics system MLCask. We define and accelerate the metric-driven merge operation by pruning the pipeline search tree using reusable history records and pipeline compatibility information. The effectiveness of MLCask is evaluated through an extensive study over several real-world deployment cases.
  arXiv  Detail & Related papers  (2020-10-17T13:34:48Z)
• AVATAR -- Machine Learning Pipeline Evaluation Using Surrogate Model [10.83607599315401]
  We propose a novel method to evaluate the validity of ML pipelines using a surrogate model (AVATAR) Our experiments show that the AVATAR is more efficient in evaluating complex pipelines in comparison with the traditional evaluation approaches requiring their execution.
  arXiv  Detail & Related papers  (2020-01-30T02:53:29Z)
• Rethinking Learning-based Demosaicing, Denoising, and Super-Resolution Pipeline [86.01209981642005]
  We study the effects of pipelines on the mixture problem of learning-based DN, DM, and SR, in both sequential and joint solutions. Our suggested pipeline DN$to$SR$to$DM yields consistently better performance than other sequential pipelines. We propose an end-to-end Trinity Pixel Enhancement NETwork (TENet) that achieves state-of-the-art performance for the mixture problem.
  arXiv  Detail & Related papers  (2019-05-07T13:19:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.