How Low Can You Go? The Data-Light SE Challenge

• URL: http://arxiv.org/abs/2512.13524v1
• Date: Mon, 15 Dec 2025 16:49:50 GMT
• Title: How Low Can You Go? The Data-Light SE Challenge
• Authors: Kishan Kumar Ganguly, Tim Menzies,
• Abstract summary: Much of software engineering (SE) research assumes that progress depends on massive datasets and CPU-intensives.<n>Counter-evidence presented in this paper suggests otherwise, including software configuration and performance tuning, cloud and systems optimization, project and process-level decision modeling, behavioral analytics, financial risk modeling, project health prediction, reinforcement learning tasks, sales forecasting, and software testing.<n>Our results highlight that some SE tasks may be better served by lightweight approaches that demand fewer labels and far less computation.
• Score: 4.282746516699565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Much of software engineering (SE) research assumes that progress depends on massive datasets and CPU-intensive optimizers. Yet has this assumption been rigorously tested? The counter-evidence presented in this paper suggests otherwise: across dozens of optimization problems from recent SE literature, including software configuration and performance tuning, cloud and systems optimization, project and process-level decision modeling, behavioral analytics, financial risk modeling, project health prediction, reinforcement learning tasks, sales forecasting, and software testing, even with just a few dozen labels, very simple methods (e.g. diversity sampling, a minimal Bayesian learner, or random probes) achieve near 90% of the best reported results. Further, these simple methods perform just as well as more state-of-the-the-art optimizers like SMAC, TPE, DEHB etc. While some tasks would require better outcomes and more sampling, these results seen after a few dozen samples would suffice for many engineering needs (particularly when the goal is rapid and cost-efficient guidance rather than slow and exhaustive optimization). Our results highlight that some SE tasks may be better served by lightweight approaches that demand fewer labels and far less computation. We hence propose the data-light challenge: when will a handful of labels suffice for SE tasks? To enable a large-scale investigation of this issue, we contribute (1) a mathematical formalization of labeling, (2) lightweight baseline algorithms, and (3) results on public-domain data showing the conditions under which lightweight methods excel or fail. For the purposes of open science, our scripts and data are online at https://github.com/KKGanguly/NEO .

Related papers

• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• BINGO! Simple Optimizers Win Big if Problems Collapse to a Few Buckets [12.069139819456861]
  Software engineering (SE) can be slow and complex.<n>This paper introduces the BINGO effect: a novel phenomenon where SE data collapses into a tiny fraction of possible solution "buckets"<n>We show the BINGO effect's prevalence across 39 optimization in SE problems.<n>Exploiting this, we optimize 10,000 times faster than state-of-the-art methods, with comparable effectiveness.
  arXiv  Detail & Related papers  (2025-06-04T23:13:58Z)
• Self-Supervised Prompt Optimization [19.159322848728646]
  Well-designed prompts are crucial for enhancing Large language models' (LLMs) reasoning capabilities.<n>Existing prompt optimization methods rely heavily on external references such as ground truth or by humans.<n>We propose Self-Supervised Prompt Optimization (SPO), a cost-efficient framework that discovers effective prompts for both closed and open-ended tasks.
  arXiv  Detail & Related papers  (2025-02-07T17:45:16Z)
• OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling [62.19438812624467]
  Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning.<n>We propose OptiBench, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs.
  arXiv  Detail & Related papers  (2024-07-13T13:27:57Z)
• TrackFormers: In Search of Transformer-Based Particle Tracking for the High-Luminosity LHC Era [2.9052912091435923]
  High-Energy Physics experiments are facing a multi-fold data increase with every new iteration.<n>One such step in need of an overhaul is the task of particle track reconstruction, a.k.a., tracking.<n>A Machine Learning-assisted solution is expected to provide significant improvements.
  arXiv  Detail & Related papers  (2024-07-09T18:47:25Z)
• A Declarative System for Optimizing AI Workloads [14.302404377396837]
  Palimpzest is a system that enables anyone to process AI-powered analytical queries simply by defining them in a declarative language. We describe the workload of AI-powered analytics tasks, the optimization methods that Palimpzest uses, and the prototype system itself. We show that even our simple prototype offers a range of appealing plans, including one that is 3.3x faster and 2.9x cheaper than the baseline method.
  arXiv  Detail & Related papers  (2024-05-23T15:31:18Z)
• Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs [18.242110417706]
  This work focuses on leveraging and selecting from vast, unlabeled, open data to pre-fine-tune a pre-trained language model. We show the optimality of this approach for fine-tuning tasks under certain conditions. Our proposed method is significantly faster than existing techniques, scaling to millions of samples within a single GPU hour.
  arXiv  Detail & Related papers  (2024-05-05T00:08:00Z)
• Making Large Language Models Better Data Creators [22.0882632635255]
  Large language models (LLMs) have advanced the state-of-the-art in NLP significantly. deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. We propose a unified data creation pipeline that requires only a single format example.
  arXiv  Detail & Related papers  (2023-10-31T01:08:34Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)
• Efficient Online Reinforcement Learning with Offline Data [78.92501185886569]
  We show that we can simply apply existing off-policy methods to leverage offline data when learning online. We extensively ablate these design choices, demonstrating the key factors that most affect performance. We see that correct application of these simple recommendations can provide a $mathbf2.5times$ improvement over existing approaches.
  arXiv  Detail & Related papers  (2023-02-06T17:30:22Z)
• Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning [57.163525407022966]
  Imbalanced learning is a fundamental challenge in data mining, where there is a disproportionate ratio of training samples in each class. Over-sampling is an effective technique to tackle imbalanced learning through generating synthetic samples for the minority class. We propose AutoSMOTE, an automated over-sampling algorithm that can jointly optimize different levels of decisions.
  arXiv  Detail & Related papers  (2022-08-26T04:28:01Z)
• Learning to Optimize: A Primer and A Benchmark [94.29436694770953]
  Learning to optimize (L2O) is an emerging approach that leverages machine learning to develop optimization methods. This article is poised to be the first comprehensive survey and benchmark of L2O for continuous optimization.
  arXiv  Detail & Related papers  (2021-03-23T20:46:20Z)
• How to distribute data across tasks for meta-learning? [59.608652082495624]
  We show that the optimal number of data points per task depends on the budget, but it converges to a unique constant value for large budgets. Our results suggest a simple and efficient procedure for data collection.
  arXiv  Detail & Related papers  (2021-03-15T15:38:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.