A multi-language toolkit for supporting automated checking of research outputs

• URL: http://arxiv.org/abs/2212.02935v2
• Date: Mon, 2 Sep 2024 20:06:21 GMT
• Title: A multi-language toolkit for supporting automated checking of research outputs
• Authors: Richard J. Preen, Maha Albashir, Simon Davy, Jim Smith,
• Abstract summary: acro distinguishes between: research output that is safe to publish; output that requires further analysis; and output that cannot be published because it creates substantial risk of disclosing private data. This is achieved through the use of a lightweight Python wrapper that sits over well-known analysis tools that produce outputs such as tables, plots, and statistical models.
• Score: 0.44998333629984877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This article presents the automatic checking of research outputs package acro, which assists researchers and data governance teams by automatically applying best-practice principles-based statistical disclosure control (SDC) techniques on-the-fly as researchers conduct their analyses. acro distinguishes between: research output that is safe to publish; output that requires further analysis; and output that cannot be published because it creates substantial risk of disclosing private data. This is achieved through the use of a lightweight Python wrapper that sits over well-known analysis tools that produce outputs such as tables, plots, and statistical models. This adds functionality to (i) identify potentially disclosive outputs against a range of commonly used disclosure tests; (ii) apply disclosure mitigation strategies where required; (iii) report reasons for applying SDC; and (iv) produce simple summary documents trusted research environment staff can use to streamline their workflow. The major analytical programming languages used by researchers are supported: Python, R, and Stata. The acro code and documentation are available under an MIT license at https://github.com/AI-SDC/ACRO

Related papers

• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• SyROCCo: Enhancing Systematic Reviews using Machine Learning [6.805429133535976]
  This paper explores the use of machine learning techniques to help navigate the systematic review process. The application of ML techniques to subsequent stages of a review, such as data extraction and evidence mapping, is in its infancy.
  arXiv  Detail & Related papers  (2024-06-24T11:04:43Z)
• Automatic benchmarking of large multimodal models via iterative experiment programming [71.78089106671581]
  We present APEx, the first framework for automatic benchmarking of LMMs. Given a research question expressed in natural language, APEx leverages a large language model (LLM) and a library of pre-specified tools to generate a set of experiments for the model at hand. The report drives the testing procedure: based on the current status of the investigation, APEx chooses which experiments to perform and whether the results are sufficient to draw conclusions.
  arXiv  Detail & Related papers  (2024-06-18T06:43:46Z)
• Are you still on track!? Catching LLM Task Drift with Activations [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users. We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set. We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Using Large Language Models to Enrich the Documentation of Datasets for Machine Learning [1.8270184406083445]
  We explore using large language models (LLM) and prompting strategies to automatically extract dimensions from documents. Our approach could aid data publishers and practitioners in creating machine-readable documentation. We have released an open-source tool implementing our approach and a replication package, including the experiments' code and results.
  arXiv  Detail & Related papers  (2024-04-04T10:09:28Z)
• MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization [86.61052121715689]
  MatPlotAgent is a model-agnostic framework designed to automate scientific data visualization tasks. MatPlotBench is a high-quality benchmark consisting of 100 human-verified test cases.
  arXiv  Detail & Related papers  (2024-02-18T04:28:28Z)
• PyRCA: A Library for Metric-based Root Cause Analysis [66.72542200701807]
  PyRCA is an open-source machine learning library of Root Cause Analysis (RCA) for Artificial Intelligence for IT Operations (AIOps) It provides a holistic framework to uncover the complicated metric causal dependencies and automatically locate root causes of incidents.
  arXiv  Detail & Related papers  (2023-06-20T09:55:10Z)
• Packaging code for reproducible research in the public sector [0.0]
  jtstats project consists of R and Python packages for importing, processing, and visualising large and complex datasets. Jtstats shows how domain specific packages can enable reproducible research within the public sector and beyond.
  arXiv  Detail & Related papers  (2023-05-25T16:07:24Z)
• Methodology to Create Analysis-Naive Holdout Records as well as Train and Test Records for Machine Learning Analyses in Healthcare [0.0]
  The purpose of the holdout sample is to preserve data for research studies that will be analysis-naive and randomly selected from the full dataset. The methodology suggested for creating holdouts is a modification of k-fold cross validation, which takes into account randomization and efficiently allows a three-way split.
  arXiv  Detail & Related papers  (2022-05-09T00:51:08Z)
• DRIFT: A Toolkit for Diachronic Analysis of Scientific Literature [0.7349727826230862]
  We open source DRIFT, which allows researchers to track research trends and development over the years. The analysis methods are collated from well-cited research works, with a few of our own methods added for good measure. To demonstrate the utility and efficacy of our tool, we perform a case study on the cs.CL corpus of the arXiv repository and draw inferences from the analysis methods.
  arXiv  Detail & Related papers  (2021-07-02T17:33:25Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.