OSS License Identification at Scale: A Comprehensive Dataset Using World of Code

• URL: http://arxiv.org/abs/2409.04824v2
• Date: Fri, 06 Dec 2024 15:04:07 GMT
• Title: OSS License Identification at Scale: A Comprehensive Dataset Using World of Code
• Authors: Mahmoud Jahanshahi, David Reid, Adam McDaniel, Audris Mockus,
• Abstract summary: This study presents a reusable and comprehensive dataset of open source software (OSS) licenses. We found and identified 5.5 million distinct license blobs in OSS projects. The dataset is open, providing a valuable resource for developers, researchers, and legal professionals in the OSS community.
• Score: 4.954816514146113
• License:
• Abstract: The proliferation of open source software (OSS) and different types of reuse has made it incredibly difficult to perform an essential legal and compliance task of accurate license identification within the software supply chain. This study presents a reusable and comprehensive dataset of OSS licenses, created using the World of Code (WoC) infrastructure. By scanning all files containing "license" in their file paths, and applying the approximate matching via winnowing algorithm to identify the most similar license from the SPDX list, we found and identified 5.5 million distinct license blobs in OSS projects. The dataset includes a detailed project-to-license (P2L) map with commit timestamps, enabling dynamic analysis of license adoption and changes over time. To verify the accuracy of the dataset we use stratified sampling and manual review, achieving a final accuracy of 92.08%, with precision of 87.14%, recall of 95.45%, and an F1 score of 91.11%. This dataset is intended to support a range of research and practical tasks, including the detection of license noncompliance, the investigations of license changes, study of licensing trends, and the development of compliance tools. The dataset is open, providing a valuable resource for developers, researchers, and legal professionals in the OSS community.

Related papers

• Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs [67.0310240737424]
  We introduce a novel approach to safeguard the ownership of text datasets and effectively detect unauthorized use by the RA-LLMs. Our approach preserves the original data completely unchanged while protecting it by inserting specifically designed canary documents into the IP dataset. During the detection process, unauthorized usage is identified by querying the canary documents and analyzing the responses of RA-LLMs.
  arXiv  Detail & Related papers  (2025-02-15T04:56:45Z)
• LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance [27.595354325922436]
  We introduce LicenseGPT, a fine-tuned foundation model (FM) specifically designed for dataset license compliance analysis. We evaluate existing legal FMs and find that the best-performing model achieves a Prediction Agreement (PA) of only 43.75%. We demonstrate that LicenseGPT reduces analysis time by 94.44%, from 108 seconds to 6 seconds per license, without compromising accuracy.
  arXiv  Detail & Related papers  (2024-12-30T19:04:13Z)
• Decorrelating Structure via Adapters Makes Ensemble Learning Practical for Semi-supervised Learning [50.868594148443215]
  In computer vision, traditional ensemble learning methods exhibit either a low training efficiency or the limited performance. We propose a lightweight, loss-function-free, and architecture-agnostic ensemble learning by the Decorrelating Structure via Adapters (DSA) for various visual tasks.
  arXiv  Detail & Related papers  (2024-08-08T01:31:38Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• Catch the Butterfly: Peeking into the Terms and Conflicts among SPDX Licenses [16.948633594354412]
  Third-party libraries (TPLs) in software development has accelerated the creation of modern software. Developers may inadvertently violate the licenses of TPLs, leading to legal issues. There is a need for a high-quality license dataset that encompasses a broad range of mainstream licenses.
  arXiv  Detail & Related papers  (2024-01-19T11:27:34Z)
• LiSum: Open Source Software License Summarization with Multi-Task Learning [16.521420821183995]
  Open source software (OSS) licenses regulate the conditions under which users can reuse, modify, and distribute the software legally. There exist various OSS licenses in the community, written in a formal language, which are typically long and complicated to understand. Motivated by the user study and the fast growth of licenses in the community, we propose the first study towards automated license summarization.
  arXiv  Detail & Related papers  (2023-09-10T16:43:51Z)
• The Software Heritage License Dataset (2022 Edition) [0.0]
  The dataset consists of 6.9 million unique license files. Additional metadata about shipped license files is also provided. The dataset can be used to conduct empirical studies on open source licensing, training of automated license cryptographics, natural language processing (NLP) analyses of legal texts.
  arXiv  Detail & Related papers  (2023-08-22T08:01:07Z)
• Black-box Dataset Ownership Verification via Backdoor Watermarking [67.69308278379957]
  We formulate the protection of released datasets as verifying whether they are adopted for training a (suspicious) third-party model. We propose to embed external patterns via backdoor watermarking for the ownership verification to protect them. Specifically, we exploit poison-only backdoor attacks ($e.g.$, BadNets) for dataset watermarking and design a hypothesis-test-guided method for dataset verification.
  arXiv  Detail & Related papers  (2022-08-04T05:32:20Z)
• Extending the WILDS Benchmark for Unsupervised Adaptation [186.90399201508953]
  We present the WILDS 2.0 update, which extends 8 of the 10 datasets in the WILDS benchmark of distribution shifts to include curated unlabeled data. These datasets span a wide range of applications (from histology to wildlife conservation), tasks (classification, regression, and detection), and modalities. We systematically benchmark state-of-the-art methods that leverage unlabeled data, including domain-invariant, self-training, and self-supervised methods.
  arXiv  Detail & Related papers  (2021-12-09T18:32:38Z)
• Can I use this publicly available dataset to build commercial AI software? Most likely not [8.853674186565934]
  We propose a new approach to assess the potential license compliance violations if a given publicly available dataset were to be used for building commercial AI software. Our results show that there are risks of license violations on 5 of these 6 studied datasets if they were used for commercial purposes.
  arXiv  Detail & Related papers  (2021-11-03T17:44:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.