Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline

• URL: http://arxiv.org/abs/2406.01055v1
• Date: Mon, 3 Jun 2024 07:09:15 GMT
• Title: Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline
• Authors: Julian Frattini, Lloyd Montgomery, Davide Fucci, Michael Unterkalmsteiner, Daniel Mendez, Jannik Fischbach,
• Abstract summary: We aim to improve the availability of research artifacts in requirements quality research. We extend an artifact recovery initiative and empirically evaluate the reasons for artifact unavailability. We compile a concise guideline for open science artifact disclosure.
• Score: 3.91424340393661
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Requirements quality research, which is dedicated to assessing and improving the quality of requirements specifications, is dependent on research artifacts like data sets (containing information about quality defects) and implementations (automatically detecting and removing these defects). However, recent research exposed that the majority of these research artifacts have become unavailable or have never been disclosed, which inhibits progress in the research domain. In this work, we aim to improve the availability of research artifacts in requirements quality research. To this end, we (1) extend an artifact recovery initiative, (2) empirically evaluate the reasons for artifact unavailability using Bayesian data analysis, and (3) compile a concise guideline for open science artifact disclosure. Our results include 10 recovered data sets and 7 recovered implementations, empirical support for artifact availability improving over time and the positive effect of public hosting services, and a pragmatic artifact management guideline open for community comments. With this work, we hope to encourage and support adherence to open science principles and improve the availability of research artifacts for the requirements research quality community.

Related papers

• AI4Research: A Survey of Artificial Intelligence for Scientific Research [55.5452803680643]
  We present a comprehensive survey on AI for Research (AI4Research)<n>We first introduce a systematic taxonomy to classify five mainstream tasks in AI4Research.<n>We identify key research gaps and highlight promising future directions.
  arXiv  Detail & Related papers  (2025-07-02T17:19:20Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts [10.225194259153426]
  Clarifying the research framing of NLP artefacts is crucial to aligning research with practical applications.<n>Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts.<n>We develop a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning.
  arXiv  Detail & Related papers  (2025-05-24T12:46:26Z)
• Research Artifacts in Secondary Studies: A Systematic Mapping in Software Engineering [0.9421843976231371]
  Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE)<n>We examined 537 secondary studies published between 2013 and 2023 to analyze the availability and reporting of research artifacts.
  arXiv  Detail & Related papers  (2025-04-17T05:11:39Z)
• Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation [58.064940977804596]
  A plethora of new AI models and tools has been proposed, promising to empower researchers and academics worldwide to conduct their research more effectively and efficiently. Ethical concerns regarding shortcomings of these tools and potential for misuse take a particularly prominent place in our discussion.
  arXiv  Detail & Related papers  (2025-02-07T18:26:45Z)
• ResearchArena: Benchmarking LLMs' Ability to Collect and Organize Information as Research Agents [21.17856299966841]
  Large language models (LLMs) have exhibited remarkable performance across various tasks in natural language processing. We develop ResearchArena, a benchmark that measures LLM agents' ability to conduct academic surveys.
  arXiv  Detail & Related papers  (2024-06-13T03:26:30Z)
• FAIRSECO: An Extensible Framework for Impact Measurement of Research Software [1.549241498953151]
  Existing methods for crediting research software and Research Software Engineers have proven to be insufficient. We have developed FAIRSECO, an open source framework with the objective of assessing the impact of research software in research through the evaluation of various factors.
  arXiv  Detail & Related papers  (2024-06-04T15:22:48Z)
• Privacy Impact Assessments in the Wild: A Scoping Review [1.7677916783208343]
  Privacy Impact Assessments (PIAs) offer a systematic process for assessing the privacy impacts of a project or system. PIAs are heralded as one of the main approaches to privacy by design, supporting the early identification of threats and controls. There is still a significant need for more primary research on the topic, both qualitative and quantitative.
  arXiv  Detail & Related papers  (2024-02-17T05:07:10Z)
• Repeatability, Reproducibility, Replicability, Reusability (4R) in Journals' Policies and Software/Data Management in Scientific Publications: A Survey, Discussion, and Perspectives [1.446375009535228]
  We have found a large gap between the citation-oriented practices, journal policies, recommendations, artifact Description/Evaluation guidelines, submission guides, technological evolution. The relationship between authors and scientific journals in their mutual efforts to jointly improve scientific results is analyzed. We propose recommendations for the journal policies, as well as a unified and standardized Reproducibility Guide for the submission of scientific articles for authors.
  arXiv  Detail & Related papers  (2023-12-18T09:02:28Z)
• Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering [0.3277163122167433]
  Empirical research in requirements engineering (RE) is constantly evolving. The underlying problem is the unavailability of data from earlier works. We examine the use of the Open Research Knowledge Graph (ORKG) as such an infrastructure to build and publish an initial Knowledge Graph of Empirical research in RE.
  arXiv  Detail & Related papers  (2023-06-29T08:55:35Z)
• GLUECons: A Generic Benchmark for Learning Under Constraints [102.78051169725455]
  In this work, we create a benchmark that is a collection of nine tasks in the domains of natural language processing and computer vision. We model external knowledge as constraints, specify the sources of the constraints for each task, and implement various models that use these constraints.
  arXiv  Detail & Related papers  (2023-02-16T16:45:36Z)
• Navigating the reporting guideline environment for computational pathology: A review [0.685316573653194]
  The aim of this work is to highlight resources and reporting guidelines available to researchers working in computational pathology. Items were compiled to create a summary for easy identification of useful resources and guidance. Over 70 published resources applicable to pathology AI research were identified.
  arXiv  Detail & Related papers  (2023-01-03T23:17:51Z)
• SciFact-Open: Towards open-domain scientific claim verification [61.288725621156864]
  We present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems. We collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1.
  arXiv  Detail & Related papers  (2022-10-25T05:45:00Z)
• Research Trends and Applications of Data Augmentation Algorithms [77.34726150561087]
  We identify the main areas of application of data augmentation algorithms, the types of algorithms used, significant research trends, their progression over time and research gaps in data augmentation literature. We expect readers to understand the potential of data augmentation, as well as identify future research directions and open questions within data augmentation research.
  arXiv  Detail & Related papers  (2022-07-18T11:38:32Z)
• Algorithmic Fairness Datasets: the Story so Far [68.45921483094705]
  Data-driven algorithms are studied in diverse domains to support critical decisions, directly impacting people's well-being. A growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair Machine Learning hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity)
  arXiv  Detail & Related papers  (2022-02-03T17:25:46Z)
• Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL Rolling Review and Beyond [58.71736531356398]
  We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow. We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
  arXiv  Detail & Related papers  (2022-01-27T11:02:43Z)
• Scaling up Search Engine Audits: Practical Insights for Algorithm Auditing [68.8204255655161]
  We set up experiments for eight search engines with hundreds of virtual agents placed in different regions. We demonstrate the successful performance of our research infrastructure across multiple data collections. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time.
  arXiv  Detail & Related papers  (2021-06-10T15:49:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.