The Ground Is Shifting: A Reflection on the Foundations of Software Measurement

• URL: http://arxiv.org/abs/2608.03007v1
• Date: Tue, 04 Aug 2026 01:41:07 GMT
• Title: The Ground Is Shifting: A Reflection on the Foundations of Software Measurement
• Abstract summary: For most of the past six decades, software measurement relied on labor-intensive data.<n>The shift to open-source tools dramatically expanded proprietary data collection techniques.
• Score: 10.346638444290413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For most of the past six decades, software measurement relied on labor-intensive manual collection of proprietary data, which hampered progress. The shift to repurposing traces from version control and related tools dramatically expanded data availability$\unicode{x2014}$especially with the rise of open-source software$\unicode{x2014}$but hinged on an often unstated assumption: that these tools are used by professional developers to build genuine software systems. However, as trace-generating tools, data types and scale, and empirical methods have all evolved, it has become clear that changes in data generation and analytical approaches affect many prior findings about software development, maintenance, and evolution. With AI agents now actively using these same tools, the resulting traces frequently violate the original assumption of human origin. To preserve the relevance of software measurement research, immediate action is needed: We must detect when foundational assumptions are violated in contemporary data and develop new methodologies that remain valid under changed circumstances. To this end, we propose a systematic AI-assisted replication program that revisits key findings using modern techniques, aiming for methods that yield consistent results on current data to keep software measurement meaningful.

Related papers

• On Automated and Explainable Provenance of AI-Generated Code [56.818202904805446]
  We argue that the next generation of CodeGenAI tools must be built on a foundation of explainable provenance.<n>We describe a research program combining large-scale empirical studies with post-hoc causal and interpretability techniques.
  arXiv  Detail & Related papers  (2026-08-03T14:50:31Z)
• SciToolAgent-Evo: An Ontology-Aware Self-Evolving Agent for Open-World Scientific Tool Acquisition [53.36281538053445]
  We propose SciToolAgent-Evo, an ontology-aware self-evolving agent for open-world scientific tool acquisition.<n> Driven by an evolving memory of skills, experiences, and an ontologized tool graph, it distills generalizable knowledge from contrastive trajectories during accumulation.<n>Once a novel tool is acquired, its scientific knowledge is completed online for seamless integration into the known graph.
  arXiv  Detail & Related papers  (2026-07-30T08:47:11Z)
• VESTA: Visual Exploration with Statistical Tool Agents [56.76643750765825]
  We introduce VESTA: Visual Exploration with Statistical Tool Agents.<n>We show that VESTA's dynamic tool creation outperforms prior agentic pipelines.<n>We also introduce DAWN, a benchmark targeting distribution fitting and time series.
  arXiv  Detail & Related papers  (2026-05-29T21:57:37Z)
• Llama-based source code vulnerability detection: Prompt engineering vs Fine tuning [0.6588840794922407]
  Large Language Models (LLMs) are considered among the most performant AI models to date.<n>We study their performance and apply different state-of-the-art techniques to enhance their effectiveness.<n>We leverage the recent open-source Llama-3.1 8B, with source code samples extracted from BigVul and PrimeVul datasets.
  arXiv  Detail & Related papers  (2025-12-09T12:08:24Z)
• Agents of Discovery [17.016402322416994]
  Large language models (LLMs) can jointly solve data analysis-based research problems.<n>We consider the task of anomaly detection via the publicly available and highly-studied LHC Olympics dataset.<n>The best agent-created solutions mirror the performance of human state-of-the-art results.
  arXiv  Detail & Related papers  (2025-09-10T12:25:13Z)
• Does Machine Unlearning Truly Remove Knowledge? [80.83986295685128]
  We introduce a comprehensive auditing framework for unlearning evaluation comprising three benchmark datasets, six unlearning algorithms, and five prompt-based auditing methods.<n>We evaluate the effectiveness and robustness of different unlearning strategies.
  arXiv  Detail & Related papers  (2025-05-29T09:19:07Z)
• Position: All Current Generative Fidelity and Diversity Metrics are Flawed [58.815519650465774]
  We show that all current generative fidelity and diversity metrics are flawed.<n>Our aim is to convince the research community to spend more effort in developing metrics, instead of models.
  arXiv  Detail & Related papers  (2025-05-28T15:10:33Z)
• Quantifying Process Quality: The Role of Effective Organizational Learning in Software Evolution [0.0]
  Real-world software applications must constantly evolve to remain relevant. Traditional methods of software quality control involve software quality models and continuous code inspection tools. However, there is a strong correlation and causation between the quality of the development process and the resulting software product.
  arXiv  Detail & Related papers  (2023-05-29T12:57:14Z)
• Applying Machine Learning Analysis for Software Quality Test [0.0]
  It is critical to comprehend what triggers maintenance and if it may be predicted. Numerous methods of assessing the complexity of created programs may produce useful prediction models. In this paper, the machine learning is applied on the available data to calculate the cumulative software failure levels.
  arXiv  Detail & Related papers  (2023-05-16T06:10:54Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.