Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

• URL: http://arxiv.org/abs/2607.04614v1
• Date: Mon, 06 Jul 2026 02:44:52 GMT
• Title: Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs
• Abstract summary: We conduct a large-scale analysis of 10,000 GitHub repositories across ten programming language ecosystems.<n>We find a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present.<n>We also find that component version and license information availability is highly dependent on the programming ecosystem.
• Score: 1.068768727414769
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern software development relies heavily on open-source components. Reusing components accelerates innovation but increases exposure to supply-chain attacks exploiting known vulnerabilities. Software Bills of Materials (SBOMs) improve software supply chain transparency by enumerating components, their versions, and their provenance. GitHub, the largest open-source development hosting platform, now automatically generates SBOMs for repositories, providing valuable metadata for risk assessment. Yet, it is unclear whether GitHub SBOMs can serve as a reliable source for vulnerability and license analysis, and how incomplete or inconsistent metadata may affect different programming ecosystems. To address this, we conduct a large-scale analysis of 10,000 GitHub repositories across ten programming language ecosystems, evaluating GitHub SBOMs against three other popular SBOM generators: Syft, Trivy, and the Microsoft SBOM Tool. Our study finds a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present. We also find that component version and license information availability is highly dependent on the programming ecosystem. Compared with the other three tools, GitHub yields results similar to the Microsoft SBOM Tool and often outperforms Syft and Trivy in providing version and license information. Finally, we discuss potential shortcomings of the GitHub SBOM Tool, directly related to how each ecosystem manages its dependencies.

Related papers

• SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers [55.39407031861402]
  We introduce SCDBench, a dataset and benchmark methodology for smart contract decompilation.<n>The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints.<n>We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning.
  arXiv  Detail & Related papers  (2026-05-27T20:08:47Z)
• What's Inside a GitHub Repository? An Empirical Study on the Contents of 10K Projects [1.439946676159516]
  This paper presents an initial empirical analysis to better understand the contents of real-world GitHub.<n>We analyze the files, directories, and extensions present in 10,000 GitHub repositories, as well as their evolution over ten years.
  arXiv  Detail & Related papers  (2026-05-15T23:30:44Z)
• Weaponizing the Commons: A Taxonomy and Detection Framework of Abuse on GitHub [31.618034402766018]
  GitHub plays a critical role in modern software supply chains, making its security an important research concern.<n>Existing studies have primarily focused on CI/CD automation, collaboration patterns, and community management, while abuse behaviors on GitHub have received little systematic investigation.<n>In this paper, we systematically review and summarize reported GitHub abuse behaviors and conduct an empirical analysis of publicly available abuse cases.<n>Based on this investigation, we propose a comprehensive taxonomy that characterizes their diverse symptoms and root causes from a software security perspective.
  arXiv  Detail & Related papers  (2026-04-20T07:37:03Z)
• A Large Scale Empirical Analysis on the Adherence Gap between Standards and Tools in SBOM [54.38424417079265]
  A Software Bill of Materials (SBOM) is a machine-readable artifact that organizes software information.<n>Following standards, organizations have developed tools for generating and utilizing SBOMs.<n>This paper presents the first large-scale, two-stage empirical analysis of the adherence gap, using our automated evaluation framework, SAP.
  arXiv  Detail & Related papers  (2026-01-09T08:26:05Z)
• Trace: Securing Smart Contract Repository Against Access Control Vulnerability [58.02691083789239]
  GitHub hosts numerous smart contract repositories containing source code, documentation, and configuration files.<n>Third-party developers often reference, reuse, or fork code from these repositories during custom development.<n>Existing tools for detecting smart contract vulnerabilities are limited in their ability to handle complex repositories.
  arXiv  Detail & Related papers  (2025-10-22T05:18:28Z)
• Policy-driven Software Bill of Materials on GitHub: An Empirical Study [14.398115591070727]
  The Software Bill of Materials (SBOM) is a machine-readable list of all the software dependencies included in a software.<n>Despite mandates from governments to use SBOM, research on this artifact is still in its early stages.
  arXiv  Detail & Related papers  (2025-09-01T08:45:39Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution [56.9361004704428]
  Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks.<n>SWE-Fixer is a novel open-source framework designed to effectively and efficiently resolve GitHub issues.<n>We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models.
  arXiv  Detail & Related papers  (2025-01-09T07:54:24Z)
• SBOM Generation Tools in the Python Ecosystem: an In-Detail Analysis [2.828503885204035]
  We analyze four popular SBOM generation tools using the CycloneDX standard. We highlight issues related to dependency versions, metadata files, remote dependencies, and optional dependencies. We identify a systematic issue with the lack of standards for metadata in the PyPI ecosystem.
  arXiv  Detail & Related papers  (2024-09-02T12:48:10Z)
• Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration [64.19431011897515]
  This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution.<n>Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy.<n>In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9% of in-house issues faced by development engineers, and solved 43.3% of problems after manual intervention.
  arXiv  Detail & Related papers  (2024-06-03T15:20:06Z)
• MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution [47.850418420195304]
  Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving GitHub issues. We propose a novel Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution.
  arXiv  Detail & Related papers  (2024-03-26T17:57:57Z)
• On the Security Blind Spots of Software Composition Analysis [46.1389163921338]
  We present a novel approach to detect vulnerable clones in the Maven repository. We retrieve over 53k potential vulnerable clones from Maven Central. We detect 727 confirmed vulnerable clones and synthesize a testable proof-of-vulnerability project for each of those.
  arXiv  Detail & Related papers  (2023-06-08T20:14:46Z)
• GitRank: A Framework to Rank GitHub Repositories [0.0]
  Open-source repositories provide wealth of information and are increasingly being used to build artificial intelligence (AI) based systems. In this hackathon, we utilize known code quality measures and GrimoireLab toolkit to implement a framework, named GitRank, to rank open-source repositories on three different criteria.
  arXiv  Detail & Related papers  (2022-05-04T23:42:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.