Predicting Software Performance with Divide-and-Learn

• URL: http://arxiv.org/abs/2306.06651v4
• Date: Sun, 4 Feb 2024 00:24:24 GMT
• Title: Predicting Software Performance with Divide-and-Learn
• Authors: Jingzhi Gong, Tao Chen
• Abstract summary: We propose an approach based on the concept of 'divide-and-learn', dubbed DaL. Experiment results from eight real-world systems and five sets of training data reveal that DaL performs no worse than the best counterpart on 33 out of 40 cases.
• Score: 3.635696352780227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting the performance of highly configurable software systems is the foundation for performance testing and quality assurance. To that end, recent work has been relying on machine/deep learning to model software performance. However, a crucial yet unaddressed challenge is how to cater for the sparsity inherited from the configuration landscape: the influence of configuration options (features) and the distribution of data samples are highly sparse. In this paper, we propose an approach based on the concept of 'divide-and-learn', dubbed DaL. The basic idea is that, to handle sample sparsity, we divide the samples from the configuration landscape into distant divisions, for each of which we build a regularized Deep Neural Network as the local model to deal with the feature sparsity. A newly given configuration would then be assigned to the right model of division for the final prediction. Experiment results from eight real-world systems and five sets of training data reveal that, compared with the state-of-the-art approaches, DaL performs no worse than the best counterpart on 33 out of 40 cases (within which 26 cases are significantly better) with up to 1.94x improvement on accuracy; requires fewer samples to reach the same/better accuracy; and producing acceptable training overhead. Practically, DaL also considerably improves different global models when using them as the underlying local models, which further strengthens its flexibility. To promote open science, all the data, code, and supplementary figures of this work can be accessed at our repository: https://github.com/ideas-labo/DaL.

Related papers

• Scaling LLM Inference with Optimized Sample Compute Allocation [56.524278187351925]
  We propose OSCA, an algorithm to find an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration. OSCA is also shown to be effective in agentic beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration.
  arXiv  Detail & Related papers  (2024-10-29T19:17:55Z)
• Dividable Configuration Performance Learning [4.949726352498762]
  We propose a model-agnostic and sparsity-robust framework for predicting configuration performance, dubbed DaL. DaL is based on the new paradigm of dividable learning that builds a model via "divide-and-learn"
  arXiv  Detail & Related papers  (2024-09-11T21:23:23Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient [52.2669490431145]
  PropEn is inspired by'matching', which enables implicit guidance without training a discriminator. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution.
  arXiv  Detail & Related papers  (2024-05-28T11:30:19Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Learning Defect Prediction from Unrealistic Data [57.53586547895278]
  Pretrained models of code have become popular choices for code understanding and generation tasks. Such models tend to be large and require commensurate volumes of training data. It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs. Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
  arXiv  Detail & Related papers  (2023-11-02T01:51:43Z)
• Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources [9.359395812292291]
  This paper proposes a framework called or>, which predicts model performance and supports data selection decisions based on partial samples of prospective data sources. or> significantly improves existing performance scaling approaches in terms of both accuracy of performance inference and computation costs associated with constructing the performance. Also, or> outperforms by a wide margin in data selection effectiveness compared to a range of other off-the-shelf solutions.
  arXiv  Detail & Related papers  (2023-07-05T17:33:41Z)
• Building Resilience to Out-of-Distribution Visual Data via Input Optimization and Model Finetuning [13.804184845195296]
  We propose a preprocessing model that learns to optimise input data for a specific target vision model. We investigate several out-of-distribution scenarios in the context of semantic segmentation for autonomous vehicles. We demonstrate that our approach can enable performance on such data comparable to that of a finetuned model.
  arXiv  Detail & Related papers  (2022-11-29T14:06:35Z)
• SCAI: A Spectral data Classification framework with Adaptive Inference for the IoT platform [0.0]
  We propose a Spectral data Classification framework with Adaptive Inference. Specifically, to allocate different computations for different samples while better exploiting the collaboration among different devices. To the best of our knowledge, this paper is the first attempt to conduct optimization by adaptive inference for spectral detection under the IoT platform.
  arXiv  Detail & Related papers  (2022-06-24T09:22:52Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)
• Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data [43.32085029569374]
  Federated learning algorithms aim to leverage distributed and diverse data stored at users' devices to learn a global phenomena. In the cases where the participants' data are strongly skewed (i.e., non-IID), the local models can overfit local data, leading to low performing global model. We propose a hierarchical learning system that performs Federated Gradient Descent on the user-edge layer and Federated Averaging on the edge-cloud layer.
  arXiv  Detail & Related papers  (2020-12-10T12:18:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.