Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models

• URL: http://arxiv.org/abs/2408.08148v1
• Date: Thu, 15 Aug 2024 13:33:20 GMT
• Title: Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models
• Abstract summary: During software development, developers often make numerous modifications to the software to address existing issues or implement new features. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing. We propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models.
• Score: 12.581051275141537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: During software development, developers often make numerous modifications to the software to address existing issues or implement new features. However, certain changes may inadvertently have a detrimental impact on the overall system performance. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing, such as load testing, or component-level performance testing to detect performance regressions. However, performance testing for the entire system is often expensive and time-consuming, posing challenges to adapting to the rapid release cycles common in modern DevOps practices. System-level performance testing cannot be conducted until the system is fully built and deployed. On the other hand, component-level testing focuses on isolated components, neglecting overall system performance and the impact of system workloads. In this paper, we propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models. Our approach uses local performance data to identify deviations at the component level, and then propagate these deviations to the architectural model. We then use the architectural model to predict regressions in the performance of the overall system. We evaluate our approach on two open-source benchmark systems and show that it can effectively detect end-to-end system performance regressions from local performance deviations with different intensities and under various system workloads. More importantly, our approach can detect regressions as early as in the development phase, in contrast to existing approaches that require the system to be fully built and deployed. Our approach is lightweight and can complement traditional system performance testing when testing resources are scarce.

Related papers

• Representative Dataset Generation Framework for AI-based Failure Analysis during real-time Validation of Automotive Software Systems [2.6126272668390373]
  A novel approach based on Hardware-in-the-Loop (HIL) simulation and real-time Fault Injection (FI) method is proposed.<n>The framework can generate not only sequential data, but also textual data including fault logs.
  arXiv  Detail & Related papers  (2026-08-10T14:28:42Z)
• Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems [2.9725171307292544]
  We present RAMP, a production-grounded infrastructure for assessing software engineering agents.<n>RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces.
  arXiv  Detail & Related papers  (2026-05-26T16:28:10Z)
• Dual-level Adaptation for Multi-Object Tracking: Building Test-Time Calibration from Experience and Intuition [29.371401864045115]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm to alleviate such distribution shifts.<n>This paper propose a Test-time from Experience and Intuition (TCEI) framework.<n>In this framework, the system utilizes transient memory to recall recently observed objects for rapid predictions.<n>Both confident and uncertain objects during online testing are exploited as historical priors and reflective cases.
  arXiv  Detail & Related papers  (2026-03-23T06:50:28Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• Detecting Performance-Relevant Changes in Configurable Software Systems [8.036869046810223]
  ConfFLARE estimates whether a change potentially impacts performance by identifying data-flow interactions with performance-relevant code.<n>It correctly detects performance regressions in almost all cases and identifies relevant features in all but two cases.<n>It reduces the number of configurations to be tested on average by $79%$ for synthetic and by $70%$ for real-world regression scenarios saving hours of performance testing time.
  arXiv  Detail & Related papers  (2025-11-21T14:20:08Z)
• Experience-Guided Adaptation of Inference-Time Reasoning Strategies [49.954515048847874]
  Experience-Guided Reasoner (EGuR) generates tailored strategies at inference time based on accumulated experience.<n>EGuR achieves up to 14% accuracy improvements over the strongest baselines while reducing computational costs by up to 111x.
  arXiv  Detail & Related papers  (2025-11-14T17:45:28Z)
• Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents [71.85020581835042]
  Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck.<n>Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail.<n>We introduce Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning.
  arXiv  Detail & Related papers  (2025-10-29T16:59:07Z)
• Identifying and Replicating Code Patterns Driving Performance Regressions in Software Systems [7.030339427131108]
  Performance mutation testing introduces intentional defects to measure and enhance fault-detection capabilities. A key challenge is understanding if generated mutants accurately reflect real-world performance issues. This study evaluates and extends mutation operators for performance testing.
  arXiv  Detail & Related papers  (2025-04-08T09:28:46Z)
• A Hybrid Framework for Statistical Feature Selection and Image-Based Noise-Defect Detection [55.2480439325792]
  This paper presents a hybrid framework that integrates both statistical feature selection and classification techniques to improve defect detection accuracy. We present around 55 distinguished features that are extracted from industrial images, which are then analyzed using statistical methods. By integrating these methods with flexible machine learning applications, the proposed framework improves detection accuracy and reduces false positives and misclassifications.
  arXiv  Detail & Related papers  (2024-12-11T22:12:21Z)
• Tracing Optimization for Performance Modeling and Regression Detection [15.99435412859094]
  A performance model analytically describes the relationship between the performance of a system and its runtime activities. We propose statistical approaches to reduce tracing overhead by identifying and excluding performance-insensitive code regions. Our approach is fully automated, making it ready to be used in production environments with minimal human effort.
  arXiv  Detail & Related papers  (2024-11-26T16:11:55Z)
• Experimental evaluation of architectural software performance design patterns in microservices [5.662788913145226]
  This research aims to quantify the effect of design patterns on system performance metrics. Real performance measurements are collected and compared with model-based predictions. Results demonstrate that, despite the difficulty to parameterize our benchmark system, model-based predictions are in line with real experimentation.
  arXiv  Detail & Related papers  (2024-08-20T12:21:40Z)
• A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks [1.3398445165628463]
  This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment. Our results indicate that Flink is the most stable and has one of the best fault recovery. K Kafka Streams shows suitable fault recovery performance and stability, but with higher event latency.
  arXiv  Detail & Related papers  (2024-04-09T10:49:23Z)
• Analyzing the Influence of Processor Speed and Clock Speed on Remaining Useful Life Estimation of Software Systems [0.9831489366502301]
  This research extends the analysis to assess how changes in environmental attributes, such as operating system and clock speed, affect RUL estimation in software. Findings are rigorously validated using real performance data from controlled test beds and compared with predictive model-generated data. This exploration yields actionable knowledge for software maintenance and optimization strategies.
  arXiv  Detail & Related papers  (2023-09-22T04:46:34Z)
• FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs [92.47146416628965]
  FuzzyFlow is a fault localization and test case extraction framework designed to test program optimizations. We leverage dataflow program representations to capture a fully reproducible system state and area-of-effect for optimizations. To reduce testing time, we design an algorithm for minimizing test inputs, trading off memory for recomputation.
  arXiv  Detail & Related papers  (2023-06-28T13:00:17Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• Improving the Performance of Robust Control through Event-Triggered Learning [74.57758188038375]
  We propose an event-triggered learning algorithm that decides when to learn in the face of uncertainty in the LQR problem. We demonstrate improved performance over a robust controller baseline in a numerical example.
  arXiv  Detail & Related papers  (2022-07-28T17:36:37Z)
• Performance Analysis of Deep Learning Workloads on a Composable System [0.08388591755871731]
  Composable infrastructure is defined as resources, such as compute, storage, accelerators and networking, that are shared in a pool. This paper details the design of an enterprise composable infrastructure that we have implemented and made available to our partners in the IBM Research AI Hardware Center.
  arXiv  Detail & Related papers  (2021-03-19T17:15:42Z)
• Unsupervised Domain Adaptation for Spatio-Temporal Action Localization [69.12982544509427]
  S-temporal action localization is an important problem in computer vision. We propose an end-to-end unsupervised domain adaptation algorithm. We show that significant performance gain can be achieved when spatial and temporal features are adapted separately or jointly.
  arXiv  Detail & Related papers  (2020-10-19T04:25:10Z)
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation [114.48767388174218]
  This paper presents an empirical analysis on different types of dialog systems composed of different modules in different settings. Our results show that a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels.
  arXiv  Detail & Related papers  (2020-05-15T05:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.