A Dataset of Software Bill of Materials for Evaluating SBOM Consumption Tools

• URL: http://arxiv.org/abs/2504.06880v1
• Date: Wed, 09 Apr 2025 13:35:02 GMT
• Title: A Dataset of Software Bill of Materials for Evaluating SBOM Consumption Tools
• Authors: Rio Kishimoto, Tetsuya Kanda, Yuki Manabe, Katsuro Inoue, Shi Qiu, Yoshiki Higo,
• Abstract summary: A Software Bill of Materials (SBOM) is a list of components used in software.<n> Numerous tools support software dependency management through SBOMs.<n>There is no publicly available dataset specifically designed for this purpose.<n>We present a dataset of SBOMs generated from real-world Java projects.
• Score: 6.081142345739704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A Software Bill of Materials (SBOM) is becoming an essential tool for effective software dependency management. An SBOM is a list of components used in software, including details such as component names, versions, and licenses. Using SBOMs, developers can quickly identify software components and assess whether their software depends on vulnerable libraries. Numerous tools support software dependency management through SBOMs, which can be broadly categorized into two types: tools that generate SBOMs and tools that utilize SBOMs. A substantial collection of accurate SBOMs is required to evaluate tools that utilize SBOMs. However, there is no publicly available dataset specifically designed for this purpose, and research on SBOM consumption tools remains limited. In this paper, we present a dataset of SBOMs to address this gap. The dataset we constructed comprises 46 SBOMs generated from real-world Java projects, with plans to expand it to include a broader range of projects across various programming languages. Accurate and well-structured SBOMs enable researchers to evaluate the functionality of SBOM consumption tools and identify potential issues. We collected 3,271 Java projects from GitHub and generated SBOMs for 798 of them using Maven with an open-source SBOM generation tool. These SBOMs were refined through both automatic and manual corrections to ensure accuracy, currently resulting in 46 SBOMs that comply with the SPDX Lite profile, which defines minimal requirements tailored to practical workflows in industries. This process also revealed issues with the SBOM generation tools themselves. The dataset is publicly available on Zenodo (DOI: 10.5281/zenodo.14233415).

Related papers

• Wild SBOMs: a Large-scale Dataset of Software Bills of Materials from Public Code [4.1920378271058425]
  Developers gain productivity by reusing readily available Free and Open Source Software (FOSS) components.<n>One approach to handle those difficulties is to use Software Bill of Materials (SBOMs)<n>A large scale study on SBOM practices based on SBOM files produced in the wild is still lacking.
  arXiv  Detail & Related papers  (2025-03-19T09:20:28Z)
• Software Bills of Materials in Maven Central [9.699225997570384]
  There is little knowledge about how developers distribute Software Bills of Materials (SBOMs)<n>We mine SBOMs from Maven Central to assess the extent to which developers publish SBOMs along with the artifacts.<n>We present our methodology to mine SBOMs, as well as novel insights about SBOM publication.
  arXiv  Detail & Related papers  (2025-01-23T16:56:40Z)
• Supply Chain Insecurity: The Lack of Integrity Protection in SBOM Solutions [0.0]
  The Software Bill of Materials (SBOM) is paramount in ensuring software supply chain security.<n>Under the Executive Order issued by President Biden, the adoption of the SBOM has become obligatory within the United States.<n>This work presents an in-depth and systematic investigation into the integrity of SBOMs.
  arXiv  Detail & Related papers  (2024-12-06T15:52:12Z)
• SBOM Generation Tools in the Python Ecosystem: an In-Detail Analysis [2.828503885204035]
  We analyze four popular SBOM generation tools using the CycloneDX standard. We highlight issues related to dependency versions, metadata files, remote dependencies, and optional dependencies. We identify a systematic issue with the lack of standards for metadata in the PyPI ecosystem.
  arXiv  Detail & Related papers  (2024-09-02T12:48:10Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models [31.742620965039517]
  Large Language Models (LLMs) excel in NLP tasks, but still need external tools to extend their ability. We introduce the Whether-or-not tool usage Evaluation benchmark (WTU-Eval) to assess LLMs with eleven datasets. The results of eight LLMs on WTU-Eval reveal that LLMs frequently struggle to determine tool use in general datasets. Fine-tuning Llama2-7B results in a 14% average performance improvement and a 16.8% decrease in incorrect tool usage.
  arXiv  Detail & Related papers  (2024-07-02T12:07:38Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error [54.954211216847135]
  Existing large language models (LLMs) only reach a correctness rate in the range of 30% to 60%. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE) STE orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
  arXiv  Detail & Related papers  (2024-03-07T18:50:51Z)
• EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction [56.02100384015907]
  EasyTool is a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction. It can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios.
  arXiv  Detail & Related papers  (2024-01-11T15:45:11Z)
• Good Tools are Half the Work: Tool Usage in Deep Learning Projects [5.966029067108828]
  The rising popularity of deep learning (DL) methods and techniques has invigorated interest in the topic of SE4DL (Software Engineering for Deep Learning) About 63% of the GitHub repositories we examined contained at least one conventional SE tool. Software construction tools are the most widely adopted, while the opposite applies to management and maintenance tools.
  arXiv  Detail & Related papers  (2023-10-29T19:21:33Z)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use [79.87054552116443]
  Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities.<n>We introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools.<n>We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools.
  arXiv  Detail & Related papers  (2023-10-04T19:39:26Z)
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs [104.37772295581088]
  Open-source large language models (LLMs), e.g., LLaMA, remain significantly limited in tool-use capabilities. We introduce ToolLLM, a general tool-usetuning encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning framework for tool use, which is constructed automatically using ChatGPT.
  arXiv  Detail & Related papers  (2023-07-31T15:56:53Z)
• Large Language Models as Tool Makers [85.00361145117293]
  We introduce a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving. Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving.
  arXiv  Detail & Related papers  (2023-05-26T17:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.