Related papers: Towards a Quality Indicator for Research Data publications and Research Software publications -- A vision from the Helmholtz Association

Towards a Quality Indicator for Research Data publications and Research Software publications -- A vision from the Helmholtz Association

URL: http://arxiv.org/abs/2401.08804v2
Date: Fri, 26 Jan 2024 09:31:47 GMT
Title: Towards a Quality Indicator for Research Data publications and Research Software publications -- A vision from the Helmholtz Association
Authors: Wolfgang zu Castell, Doris Dransch, Guido Juckeland, Marcel Meistring, Bernadette Fritzsch, Ronny Gey, Britta H\"opfner, Martin K\"ohler, Christian Mee{\ss}en, Hela Mehrtens, Felix M\"uhlbauer, Sirko Schindler, Thomas Schnicke, Roland Bertelmann
Abstract summary: There is not yet an established process to assess and evaluate quality of research data and research software publications. The Task Group Quality Indicators for Data and Software Publications currently develops a quality indicator for research data and research software publications.
Score: 0.24848203755267903
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Research data and software are widely accepted as an outcome of scientific work. However, in comparison to text-based publications, there is not yet an established process to assess and evaluate quality of research data and research software publications. This paper presents an attempt to fill this gap. Initiated by the Working Group Open Science of the Helmholtz Association the Task Group Helmholtz Quality Indicators for Data and Software Publications currently develops a quality indicator for research data and research software publications to be used within the Association. This report summarizes the vision of the group of what all contributes to such an indicator. The proposed approach relies on generic well-established concepts for quality criteria, such as the FAIR Principles and the COBIT Maturity Model. It does - on purpose - not limit itself to technical implementation possibilities to avoid using an existing metric for a new purpose. The intention of this paper is to share the current state for further discussion with all stakeholders, particularly with other groups also working on similar metrics but also with entities that use the metrics.

Related papers

Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
arXiv Detail & Related papers (2024-10-02T20:48:28Z)
Data-Centric AI in the Age of Large Language Models [51.20451986068925]
This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs) We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
arXiv Detail & Related papers (2024-06-20T16:34:07Z)
Leveraging Artificial Intelligence Technology for Mapping Research to Sustainable Development Goals: A Case Study [6.551575555269426]
This study employed over 82,000 publications from an Australian university as a case study. We utilized a similarity measure to map these publications onto Sustainable Development Goals. We leveraged the OpenAI GPT model to conduct the same task, facilitating a comparative analysis between the two approaches.
arXiv Detail & Related papers (2023-11-09T11:44:22Z)
Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering [0.3277163122167433]
Empirical research in requirements engineering (RE) is constantly evolving. The underlying problem is the unavailability of data from earlier works. We examine the use of the Open Research Knowledge Graph (ORKG) as such an infrastructure to build and publish an initial Knowledge Graph of Empirical research in RE.
arXiv Detail & Related papers (2023-06-29T08:55:35Z)
CoCon: A Data Set on Combined Contextualized Research Artifact Use [0.0]
CoCon is a large scholarly data set reflecting the combined use of research artifacts in academic publications' full-text. Our data set comprises 35 k artifacts (data sets, methods, models, and tasks) and 340 k publications. We formalize a link prediction task for "combined research artifact use prediction" and provide code to utilize analyses of and the development of ML applications on our data.
arXiv Detail & Related papers (2023-03-27T13:29:09Z)
Assessing Scientific Contributions in Data Sharing Spaces [64.16762375635842]
This paper introduces the SCIENCE-index, a blockchain-based metric measuring a researcher's scientific contributions. To incentivize researchers to share their data, the SCIENCE-index is augmented to include a data-sharing parameter. Our model is evaluated by comparing the distribution of its output for geographically diverse researchers to that of the h-index.
arXiv Detail & Related papers (2023-03-18T19:17:47Z)
Artificial intelligence technologies to support research assessment: A review [10.203602318836444]
This literature review identifies indicators that associate with higher impact or higher quality research from article text. It includes studies that used machine learning techniques to predict citation counts or quality scores for journal articles or conference papers.
arXiv Detail & Related papers (2022-12-11T06:58:39Z)
Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z)
Tag-Aware Document Representation for Research Paper Recommendation [68.8204255655161]
We propose a hybrid approach that leverages deep semantic representation of research papers based on social tags assigned by users. The proposed model is effective in recommending research papers even when the rating data is very sparse.
arXiv Detail & Related papers (2022-09-08T09:13:07Z)
Best Practices and Scoring System on Reviewing A.I. based Medical Imaging Papers: Part 1 Classification [0.9428556282541211]
The Machine Learning Education Sub-Committee of SIIM has identified a knowledge gap and a serious need to establish guidelines for reviewing these studies. This first entry in the series focuses on the task of image classification. The goal of this series is to provide resources to help improve the review process for A.I.-based medical imaging papers.
arXiv Detail & Related papers (2022-02-03T21:46:59Z)
Recognizing Families In the Wild: White Paper for the 4th Edition Data Challenge [91.55319616114943]
This paper summarizes the supported tasks (i.e., kinship verification, tri-subject verification, and search & retrieval of missing children) in the Recognizing Families In the Wild (RFIW) evaluation. The purpose of this paper is to describe the 2020 RFIW challenge, end-to-end, along with forecasts in promising future directions.
arXiv Detail & Related papers (2020-02-15T02:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.