OSS License Identification at Scale: A Comprehensive Dataset Using World of Code

• URL: http://arxiv.org/abs/2409.04824v1
• Date: Sat, 7 Sep 2024 13:34:55 GMT
• Title: OSS License Identification at Scale: A Comprehensive Dataset Using World of Code
• Authors: Mahmoud Jahanshahi, David Reid, Adam McDaniel, Audris Mockus,
• Abstract summary: We employ an exhaustive approach, scanning all files containing license'' in their filepath, and apply the winnowing algorithm for robust text matching. Our method identifies and matches over 5.5 million distinct license blobs across millions of OSS projects, creating a detailed project-to-license (P2L) map.
• Score: 4.954816514146113
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The proliferation of open source software (OSS) has led to a complex landscape of licensing practices, making accurate license identification crucial for legal and compliance purposes. This study presents a comprehensive analysis of OSS licenses using the World of Code (WoC) infrastructure. We employ an exhaustive approach, scanning all files containing ``license'' in their filepath, and apply the winnowing algorithm for robust text matching. Our method identifies and matches over 5.5 million distinct license blobs across millions of OSS projects, creating a detailed project-to-license (P2L) map. We verify the accuracy of our approach through stratified sampling and manual review, achieving a final accuracy of 92.08%, with precision of 87.14%, recall of 95.45%, and an F1 score of 91.11%. This work enhances the understanding of OSS licensing practices and provides a valuable resource for developers, researchers, and legal professionals. Future work will expand the scope of license detection to include code files and references to licenses in project documentation.

Related papers

• Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs [67.0310240737424]
  We introduce a novel approach to safeguard the ownership of text datasets and effectively detect unauthorized use by the RA-LLMs. Our approach preserves the original data completely unchanged while protecting it by inserting specifically designed canary documents into the IP dataset. During the detection process, unauthorized usage is identified by querying the canary documents and analyzing the responses of RA-LLMs.
  arXiv  Detail & Related papers  (2025-02-15T04:56:45Z)
• LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance [27.595354325922436]
  We introduce LicenseGPT, a fine-tuned foundation model (FM) specifically designed for dataset license compliance analysis. We evaluate existing legal FMs and find that the best-performing model achieves a Prediction Agreement (PA) of only 43.75%. We demonstrate that LicenseGPT reduces analysis time by 94.44%, from 108 seconds to 6 seconds per license, without compromising accuracy.
  arXiv  Detail & Related papers  (2024-12-30T19:04:13Z)
• Decorrelating Structure via Adapters Makes Ensemble Learning Practical for Semi-supervised Learning [50.868594148443215]
  In computer vision, traditional ensemble learning methods exhibit either a low training efficiency or the limited performance. We propose a lightweight, loss-function-free, and architecture-agnostic ensemble learning by the Decorrelating Structure via Adapters (DSA) for various visual tasks.
  arXiv  Detail & Related papers  (2024-08-08T01:31:38Z)
• Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs [60.32717556756674]
  This paper introduces a systematic evaluation framework to assess Large Language Models in detecting cryptographic misuses. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T15:31:26Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• Catch the Butterfly: Peeking into the Terms and Conflicts among SPDX Licenses [16.948633594354412]
  Third-party libraries (TPLs) in software development has accelerated the creation of modern software. Developers may inadvertently violate the licenses of TPLs, leading to legal issues. There is a need for a high-quality license dataset that encompasses a broad range of mainstream licenses.
  arXiv  Detail & Related papers  (2024-01-19T11:27:34Z)
• LiSum: Open Source Software License Summarization with Multi-Task Learning [16.521420821183995]
  Open source software (OSS) licenses regulate the conditions under which users can reuse, modify, and distribute the software legally. There exist various OSS licenses in the community, written in a formal language, which are typically long and complicated to understand. Motivated by the user study and the fast growth of licenses in the community, we propose the first study towards automated license summarization.
  arXiv  Detail & Related papers  (2023-09-10T16:43:51Z)
• The Software Heritage License Dataset (2022 Edition) [0.0]
  The dataset consists of 6.9 million unique license files. Additional metadata about shipped license files is also provided. The dataset can be used to conduct empirical studies on open source licensing, training of automated license cryptographics, natural language processing (NLP) analyses of legal texts.
  arXiv  Detail & Related papers  (2023-08-22T08:01:07Z)
• Black-box Dataset Ownership Verification via Backdoor Watermarking [67.69308278379957]
  We formulate the protection of released datasets as verifying whether they are adopted for training a (suspicious) third-party model. We propose to embed external patterns via backdoor watermarking for the ownership verification to protect them. Specifically, we exploit poison-only backdoor attacks ($e.g.$, BadNets) for dataset watermarking and design a hypothesis-test-guided method for dataset verification.
  arXiv  Detail & Related papers  (2022-08-04T05:32:20Z)
• Extending the WILDS Benchmark for Unsupervised Adaptation [186.90399201508953]
  We present the WILDS 2.0 update, which extends 8 of the 10 datasets in the WILDS benchmark of distribution shifts to include curated unlabeled data. These datasets span a wide range of applications (from histology to wildlife conservation), tasks (classification, regression, and detection), and modalities. We systematically benchmark state-of-the-art methods that leverage unlabeled data, including domain-invariant, self-training, and self-supervised methods.
  arXiv  Detail & Related papers  (2021-12-09T18:32:38Z)
• Can I use this publicly available dataset to build commercial AI software? Most likely not [8.853674186565934]
  We propose a new approach to assess the potential license compliance violations if a given publicly available dataset were to be used for building commercial AI software. Our results show that there are risks of license violations on 5 of these 6 studied datasets if they were used for commercial purposes.
  arXiv  Detail & Related papers  (2021-11-03T17:44:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.