An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes

• URL: http://arxiv.org/abs/2310.09634v1
• Date: Sat, 14 Oct 2023 18:01:11 GMT
• Title: An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes
• Authors: Ey\"up Kaan Akdeniz, Selma Tekir, Malik Nizar Asad Al Hinnawi
• Abstract summary: We propose an end-to-end system that operates on the Readme file of the source code repositories. The system generates scores based on a custom function to combine section scores. It has an advantage regarding explainability since one can directly relate the score to the sections of Readme files.
• Score: 0.138120109831448
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Increased reproducibility of machine learning research has been a driving force for dramatic improvements in learning performances. The scientific community further fosters this effort by including reproducibility ratings in reviewer forms and considering them as a crucial factor for the overall evaluation of papers. Accompanying source code is not sufficient to make a work reproducible. The shared codes should meet the ML reproducibility checklist as well. This work aims to support reproducibility evaluations of papers with source codes. We propose an end-to-end system that operates on the Readme file of the source code repositories. The system checks the compliance of a given Readme to a template proposed by a widely used platform for sharing source codes of research. Our system generates scores based on a custom function to combine section scores. We also train a hierarchical transformer model to assign a class label to a given Readme. The experimental results show that the section similarity-based system performs better than the hierarchical transformer. Moreover, it has an advantage regarding explainability since one can directly relate the score to the sections of Readme files.

Related papers

• Core: Robust Factual Precision with Informative Sub-Claim Identification [44.36892500212747]
  We observe that popular metrics can be manipulated by adding obvious or repetitive subclaims to artificially inflate scores. This observation motivates our new customizable plug-and-play subclaim selection component called Core. We show that many popular factual precision metrics augmented by Core are substantially more robust on a wide range of knowledge domains.
  arXiv  Detail & Related papers  (2024-07-04T01:51:38Z)
• Advanced Detection of Source Code Clones via an Ensemble of Unsupervised Similarity Measures [0.0]
  This research introduces a novel ensemble learning approach for code similarity assessment. The key idea is that the strengths of a diverse set of similarity measures can complement each other and mitigate individual weaknesses.
  arXiv  Detail & Related papers  (2024-05-03T13:42:49Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Data [21.247650660908484]
  We have constructed a large-scale Controllable Multimodal Feedback Synthesis dataset and propose a controllable feedback synthesis system. The system features an encoder, decoder, and controllability block for textual and visual inputs. The CMFeed dataset includes images, texts, reactions to the posts, human comments with relevance scores, and reactions to these comments. These reactions train the model to produce feedback with specified sentiments, achieving a sentiment classification accuracy of 77.23%, which is 18.82% higher than the accuracy without controllability.
  arXiv  Detail & Related papers  (2024-02-12T13:27:22Z)
• Attributable and Scalable Opinion Summarization [79.87892048285819]
  We generate abstractive summaries by decoding frequent encodings, and extractive summaries by selecting the sentences assigned to the same frequent encodings. Our method is attributable, because the model identifies sentences used to generate the summary as part of the summarization process. It scales easily to many hundreds of input reviews, because aggregation is performed in the latent space rather than over long sequences of tokens.
  arXiv  Detail & Related papers  (2023-05-19T11:30:37Z)
• RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation [96.75695811963242]
  RepoCoder is a framework to streamline the repository-level code completion process. It incorporates a similarity-based retriever and a pre-trained code language model. It consistently outperforms the vanilla retrieval-augmented code completion approach.
  arXiv  Detail & Related papers  (2023-03-22T13:54:46Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms [6.133522864509327]
  CARLA (Counterfactual And Recourse LibrAry) is a python library for benchmarking counterfactual explanation methods. We provide an extensive benchmark of 11 popular counterfactual explanation methods. We also provide a benchmarking framework for research on future counterfactual explanation methods.
  arXiv  Detail & Related papers  (2021-08-02T11:00:43Z)
• A Transformer-based Approach for Source Code Summarization [86.08359401867577]
  We learn code representation for summarization by modeling the pairwise relationship between code tokens. We show that despite the approach is simple, it outperforms the state-of-the-art techniques by a significant margin.
  arXiv  Detail & Related papers  (2020-05-01T23:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.