Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health

• URL: http://arxiv.org/abs/2301.06577v2
• Date: Wed, 11 Oct 2023 17:10:13 GMT
• Title: Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health
• Authors: Andre Lustosa, Tim Menzies
• Abstract summary: This paper only explores the application of niSNEAK to project health. That said, we see nothing in principle that prevents the application of this technique to a wider range of problems.
• Score: 13.19204187502255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When data is scarce, software analytics can make many mistakes. For example, consider learning predictors for open source project health (e.g. the number of closed pull requests in twelve months time). The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). The models generated from such tiny data sets can make many prediction errors. Those errors can be tamed by a {\em landscape analysis} that selects better learner control parameters. Our niSNEAK tool (a)~clusters the data to find the general landscape of the hyperparameters; then (b)~explores a few representatives from each part of that landscape. niSNEAK is both faster and more effective than prior state-of-the-art hyperparameter optimization algorithms (e.g. FLASH, HYPEROPT, OPTUNA). The configurations found by niSNEAK have far less error than other methods. For example, for project health indicators such as $C$= number of commits; $I$=number of closed issues, and $R$=number of closed pull requests, niSNEAK's 12 month prediction errors are \{I=0\%, R=33\%\,C=47\%\} Based on the above, we recommend landscape analytics (e.g. niSNEAK) especially when learning from very small data sets. This paper only explores the application of niSNEAK to project health. That said, we see nothing in principle that prevents the application of this technique to a wider range of problems. To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak

Related papers

• Just How Flexible are Neural Networks in Practice? [89.80474583606242]
  It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters. In practice, however, we only find solutions via our training procedure, including the gradient and regularizers, limiting flexibility.
  arXiv  Detail & Related papers  (2024-06-17T12:24:45Z)
• Streamlining Software Reviews: Efficient Predictive Modeling with Minimal Examples [11.166755101891402]
  This paper proposes a new challenge problem for software analytics. In the process we shall call "software review", a panel of SMEs (subject matter experts) review examples of software behavior to recommend how to improve that's software's operation. To support this review process, we explore methods that train a predictive model to guess if some oracle will like/dislike the next example. In 31 case studies, we show that such predictive models can be built using as few as 12 to 30 labels.
  arXiv  Detail & Related papers  (2024-05-21T16:42:02Z)
• Is Hyper-Parameter Optimization Different for Software Analytics? [11.85735565104864]
  SE data can have "smoother" boundaries between classes. SMOOTHIE runs faster and predicts better on the SE data--but ties on non-SE data with the AI tool.
  arXiv  Detail & Related papers  (2024-01-17T22:23:29Z)
• Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking [66.83273589348758]
  Link prediction attempts to predict whether an unseen edge exists based on only a portion of edges of a graph. A flurry of methods have been introduced in recent years that attempt to make use of graph neural networks (GNNs) for this task. New and diverse datasets have also been created to better evaluate the effectiveness of these new models.
  arXiv  Detail & Related papers  (2023-06-18T01:58:59Z)
• How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench [52.11481619456093]
  We study the performance prediction problem on experiment records from BIG-bench. An $R2$ score greater than 95% indicates the presence of learnable patterns within the experiment records. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being $3times$ smaller.
  arXiv  Detail & Related papers  (2023-05-24T09:35:34Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• When Less is More: On the Value of "Co-training" for Semi-Supervised Software Defect Predictors [15.862838836160634]
  This paper applies a wide range of 55 semi-supervised learners to over 714 projects. We find that semi-supervised "co-training methods" work significantly better than other approaches.
  arXiv  Detail & Related papers  (2022-11-10T23:39:12Z)
• The Early Bird Catches the Worm: Better Early Life Cycle Defect Predictors [23.22715542777918]
  In 240 GitHub projects, we find that the information in that data clumps'' towards the earliest parts of the project. A defect prediction model learned from just the first 150 commits works as well, or better than state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2021-05-24T03:49:09Z)
• How to distribute data across tasks for meta-learning? [59.608652082495624]
  We show that the optimal number of data points per task depends on the budget, but it converges to a unique constant value for large budgets. Our results suggest a simple and efficient procedure for data collection.
  arXiv  Detail & Related papers  (2021-03-15T15:38:47Z)
• Injecting Knowledge in Data-driven Vehicle Trajectory Predictors [82.91398970736391]
  Vehicle trajectory prediction tasks have been commonly tackled from two perspectives: knowledge-driven or data-driven. In this paper, we propose to learn a "Realistic Residual Block" (RRB) which effectively connects these two perspectives. Our proposed method outputs realistic predictions by confining the residual range and taking into account its uncertainty.
  arXiv  Detail & Related papers  (2021-03-08T16:03:09Z)
• Early Life Cycle Software Defect Prediction. Why? How? [37.48549087467758]
  We analyzed hundreds of popular GitHub projects for 84 months. Across these projects, most of the defects occur very early in their life cycle. We hope these results inspire other researchers to adopt a "simplicity-first" approach to their work.
  arXiv  Detail & Related papers  (2020-11-26T00:13:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.