Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

• URL: http://arxiv.org/abs/2608.28396v1
• Date: Fri, 28 Aug 2026 14:49:59 GMT
• Title: Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study
• Abstract summary: We present an empirical study of the open-source robustness tooling ecosystem.<n>Our results show that engagement and maintenance are unevenly distributed.<n>These findings highlight the need to treat robustness tools as evolving software systems.
• Score: 0.19999259391104388
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robustness evaluation is essential for deploying machine-learning (ML) systems in real-world settings, where models may face adversarial perturbations, distribution shifts, and other operational stressors. Many open-source tools, including Adversarial Robustness Toolbox, Foolbox, and Robustness Gym, support robustness testing and evaluation. However, little is known about how these tools are maintained, publicly engaged with, and sustained over time, even though practitioners may rely on them to select evaluation dependencies, reproduce robustness assessments, and provide evidence for AI assurance. We present an empirical study of the open-source robustness tooling ecosystem. Starting from a curated seed set derived from prior work, we systematically searched GitHub and identified 28 robustness-tool repositories. We analyzed repository artifacts to characterize observable community engagement, maintenance activity, and project longevity using established software-engineering metrics. Our results show that engagement and maintenance are unevenly distributed, with sustained activity concentrated in a small subset of repositories. At the data collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived. These findings highlight the need to treat robustness tools as evolving software systems.

Related papers

• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• ToolOmni: Enabling Open-World Tool Use via Agentic learning with Proactive Retrieval and Grounded Execution [49.30293260493709]
  Large Language Models (LLMs) enhance their problem-solving capability by utilizing external tools.<n>Existing methods relying on static embedding retrieval or parameter memorization struggle to align user intent with tool semantics.<n>We present Tool Omni, a unified agentic framework that enables LLMs for open-world tool use by proactive retrieval and grounded execution.
  arXiv  Detail & Related papers  (2026-04-15T12:26:10Z)
• EvolveTool-Bench: Evaluating the Quality of LLM-Generated Tool Libraries as Software Artifacts [0.0]
  We introduce EvolveToolBench, a diagnostic benchmark for tool libraries in software engineering.<n>We define library-level software quality metrics -- reuse, redundancy, composition success, regression, and safety.<n>We show that systems with similar task completion differ by up to 18% in library health, revealing software quality risks invisible to task-only evaluation.
  arXiv  Detail & Related papers  (2026-04-01T02:21:55Z)
• Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents [22.64138018985385]
  We argue that failures stem from both tool-use accuracy (how well an agent invokes a tool) and intrinsic tool accuracy (the tool's own correctness)<n>We introduce OpenTools, a community-driven toolbox that standardizes tool schemas.<n>OpenTools includes the core framework, an initial tool set, evaluation pipelines, and a contribution protocol.
  arXiv  Detail & Related papers  (2026-03-31T18:42:36Z)
• FeatureBench: Benchmarking Agentic Coding for Complex Feature Development [42.26354337364403]
  FeatureBench is a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development.<n>It incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort.<n> Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, achieves a 74.4% resolved rate on SWE-bench.
  arXiv  Detail & Related papers  (2026-02-11T16:06:32Z)
• Forecasting the Maintained Score from the OpenSSF Scorecard for GitHub Repositories linked to PyPI libraries [78.48200143057376]
  We study to what extent future maintenance activity, as captured by the OpenSSF maintained score, can be forecasted.<n>We analyze 3,220 GitHub repositories associated with the top 1% most central PyPI libraries by PageRank.<n>Our results show that future maintenance activity can be predicted with meaningful accuracy.
  arXiv  Detail & Related papers  (2026-01-26T10:32:54Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• Predicting Abandonment of Open Source Software Projects with An Integrated Feature Framework [15.50732865972961]
  Open Source Software (OSS) is a cornerstone of contemporary software development, yet the increasing prevalence of OSS project abandonment threatens global supply chains.<n>In this work, we address these challenges by reliably detecting OSS project abandonment through a dual approach: explicit archival status and rigorous semantic analysis of project documentation or description.<n>Building on this foundation, we introduce an integrated, multi-perspective feature framework for abandonment prediction, capturing user-centric, maintainer-centric, and project evolution features.<n>Our work provides reproducible, scalable, and practitioner-oriented support for understanding and managing abandonment risk in large OSS ecosystems.
  arXiv  Detail & Related papers  (2025-07-29T10:45:24Z)
• Aurora: Are Android Malware Classifiers Reliable and Stable under Distribution Shift? [51.12297424766236]
  AURORA is a framework to evaluate malware classifiers based on their confidence quality and operational resilience.<n>AURORA is complemented by a set of metrics designed to go beyond point-in-time performance.<n>The fragility in SOTA frameworks across datasets of varying drift suggests the need for a return to the whiteboard.
  arXiv  Detail & Related papers  (2025-05-28T20:22:43Z)
• Revealing the value of Repository Centrality in lifespan prediction of Open Source Software Projects [5.438725298163702]
  We propose a novel metric from the user-repository network, and leverage the metric to fit project deprecation predictors. We establish a comprehensive dataset containing 103,354 non-fork GitHub OSS projects spanning from 2011 to 2023. Our study reveals a correlation between the HITS centrality metrics and the repository deprecation risk.
  arXiv  Detail & Related papers  (2024-05-13T07:07:54Z)
• RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems [100.54655931138444]
  We propose a more holistic view of robustness for recommender systems that encompasses multiple dimensions. We present a robustness evaluation toolkit, Robustness Gym for RecSys, that allows us to quickly and uniformly evaluate the robustness of recommender system models.
  arXiv  Detail & Related papers  (2022-01-12T10:32:53Z)
• Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification [15.35099402481255]
  Uncertainty Toolbox is a python library that helps to assess, visualize, and improve uncertainty quantification. It additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references.
  arXiv  Detail & Related papers  (2021-09-21T15:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.