PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages

• URL: http://arxiv.org/abs/2404.16565v1
• Date: Thu, 25 Apr 2024 12:27:59 GMT
• Title: PyRadar: Towards Automatically Retrieving and Validating Source Code Repository Information for PyPI Packages
• Authors: Kai Gao, Weiwei Xu, Wenhao Yang, Minghui Zhou,
• Abstract summary: Existing tools can only retrieve repository information for up to 70.5% of PyPI releases. This paper proposes PyRadar, a novel framework that utilizes the metadata and source distribution to retrieve and validate the repository information for PyPI releases.
• Score: 24.8919191161202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A package's source code repository records the development history of the package, providing indispensable information for the use and risk monitoring of the package. However, a package release often misses its source code repository due to the separation of the package's development platform from its distribution platform. Existing tools retrieve the release's repository information from its metadata, which suffers from two limitations: the metadata may not contain or contain wrong information. Our analysis shows that existing tools can only retrieve repository information for up to 70.5% of PyPI releases. To address the limitations, this paper proposes PyRadar, a novel framework that utilizes the metadata and source distribution to retrieve and validate the repository information for PyPI releases. We start with an empirical study to compare four existing tools on 4,227,425 PyPI releases and analyze phantom files (files appearing in the release's distribution but not in the release's repository) in 14,375 correct package-repository links and 2,064 incorrect links. Based on the findings, we design PyRadar with three components, i.e., Metadata-based Retriever, Source Code Repository Validator, and Source Code-based Retriever. In particular, the Metadata-based Retriever combines best practices of existing tools and successfully retrieves repository information from the metadata for 72.1% of PyPI releases. The Source Code Repository Validator applies common machine learning algorithms on six crafted features and achieves an AUC of up to 0.995. The Source Code-based Retriever queries World of Code with the SHA-1 hashes of all Python files in the release's source distribution and retrieves repository information for 90.2% of packages in our dataset with an accuracy of 0.970. Both practitioners and researchers can employ the PyRadar to better use PyPI packages.

Related papers

• Analyzing the Accessibility of GitHub Repositories for PyPI and NPM Libraries [91.97201077607862]
  Industrial applications heavily rely on open-source software (OSS) libraries, which provide various benefits. To monitor the activities of such communities, a comprehensive list of repositories for the libraries of an ecosystem must be accessible. In this study, we analyze the accessibility of GitHub repositories for PyPI and NPM libraries.
  arXiv  Detail & Related papers  (2024-04-26T13:27:04Z)
• Less is More? An Empirical Study on Configuration Issues in Python PyPI Ecosystem [38.44692482370243]
  Python is widely used in the open-source community, largely owing to the extensive support from diverse third-party libraries. Third-party libraries can potentially lead to conflicts in dependencies, prompting researchers to develop dependency conflict detectors. endeavors have been made to automatically infer dependencies.
  arXiv  Detail & Related papers  (2023-10-19T09:07:51Z)
• The Stackage Repository: An Exploratory Study of its Evolution [0.0]
  This paper conducts empirical research about the evolution of Stackage considering monad packages. To the best of our knowledge, this is the first large-scale analysis of the evolution of the Stackage repository regarding packages used and monads.
  arXiv  Detail & Related papers  (2023-10-16T23:42:47Z)
• On the Feasibility of Cross-Language Detection of Malicious Packages in npm and PyPI [6.935278888313423]
  Malicious users started to spread malware by publishing open-source packages containing malicious code. Recent works apply machine learning techniques to detect malicious packages in the npm ecosystem. We present a novel approach that involves a set of language-independent features and the training of models capable of detecting malicious packages in npm and PyPI.
  arXiv  Detail & Related papers  (2023-10-14T12:32:51Z)
• scikit-fda: A Python Package for Functional Data Analysis [0.0]
  scikit-fda is a Python package for Functional Data Analysis (FDA) It provides a comprehensive set of tools for representation, preprocessing, and exploratory analysis of functional data.
  arXiv  Detail & Related papers  (2022-11-04T16:34:03Z)
• DADApy: Distance-based Analysis of DAta-manifolds in Python [51.37841707191944]
  DADApy is a python software package for analysing and characterising high-dimensional data. It provides methods for estimating the intrinsic dimension and the probability density, for performing density-based clustering and for comparing different distance metrics.
  arXiv  Detail & Related papers  (2022-05-04T08:41:59Z)
• Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Code [74.28810048824519]
  Repro is an open-source library which aims at improving the usability of research code. It provides a lightweight Python API for running software released by researchers within Docker containers.
  arXiv  Detail & Related papers  (2022-04-29T01:54:54Z)
• PyGOD: A Python Library for Graph Outlier Detection [56.33769221859135]
  PyGOD is an open-source library for detecting outliers in graph data. It supports a wide array of leading graph-based methods for outlier detection. PyGOD is released under a BSD 2-Clause license at https://pygod.org and at the Python Package Index (PyPI)
  arXiv  Detail & Related papers  (2022-04-26T06:15:21Z)
• PyHHMM: A Python Library for Heterogeneous Hidden Markov Models [63.01207205641885]
  PyHHMM is an object-oriented Python implementation of Heterogeneous-Hidden Markov Models (HHMMs) PyHHMM emphasizes features not supported in similar available frameworks: a heterogeneous observation model, missing data inference, different model order selection criterias, and semi-supervised training. PyHHMM relies on the numpy, scipy, scikit-learn, and seaborn Python packages, and is distributed under the Apache-2.0 License.
  arXiv  Detail & Related papers  (2022-01-12T07:32:36Z)
• mvlearn: Multiview Machine Learning in Python [103.55817158943866]
  mvlearn is a Python library which implements the leading multiview machine learning methods. The package can be installed from Python Package Index (PyPI) and the conda package manager.
  arXiv  Detail & Related papers  (2020-05-25T02:35:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.