Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP

• URL: http://arxiv.org/abs/2109.06598v1
• Date: Tue, 14 Sep 2021 11:36:42 GMT
• Title: Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP
• Abstract summary: This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research.
• Score: 6.3596637237946725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key part of the NLP ethics movement is responsible use of data, but exactly what that means or how it can be best achieved remain unclear. This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research across the community. Our proposal aims to contribute to the development of a consistent standard for data (re-)use, embraced across NLP conferences.

Related papers

• Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025 [3.5138213455573095]
  ACL released the EMNLP 2025 Checklists to aid transparency on the current research practice, which we focus on.<n>We provide the first analysis of recent EMNLP Checklists, by examining $73,922$ responses and justifications to them.<n>For the Main track, we find that authors isolate ethics questions of the Checklist from the paper's bulk, mimicking the trend of ethics being an afterthought.<n>We find significant issues with the checklist design and effort of authors, namely that $6%$ of all checklists contained logical contradictions between parent and child responses.
  arXiv  Detail & Related papers  (2026-08-10T08:35:26Z)
• ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses [5.170485483791494]
  ARR Responsible NLP Research checklist website states that the "checklist is designed to encourage best practices for responsible research"<n>We introduce ConfReady, a retrieval-augmented generation (RAG) application that can be used to empower authors to reflect on their work and assist authors with conference checklists.<n>To evaluate checklist assistants, we curate a dataset of 1,975 ACL checklist responses, analyze problems in human answers, and benchmark RAG and Large Language Model (LM) based systems on an evaluation subset.
  arXiv  Detail & Related papers  (2024-08-07T21:07:13Z)
• What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
  In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
  arXiv  Detail & Related papers  (2024-05-10T16:06:43Z)
• SoK: The Gap Between Data Rights Ideals and Reality [46.14715472341707]
  Do rights-based privacy laws effectively empower individuals over their data? This paper scrutinizes these approaches by reviewing empirical studies, news articles, and blog posts.
  arXiv  Detail & Related papers  (2023-12-03T21:52:51Z)
• Reproducibility in NLP: What Have We Learned from the Checklist? [48.86370747121637]
  The *CL conferences created the NLP Reproducibility Checklist in 2020 to be completed by authors at submission to remind them of key information to include. We provide the first analysis of the Checklist by examining 10,405 anonymous responses to it. We find that the 44% of submissions that gather new data are 5% less likely to be accepted than those that did not.
  arXiv  Detail & Related papers  (2023-06-16T00:39:25Z)
• NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
  We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
  arXiv  Detail & Related papers  (2022-11-12T12:29:38Z)
• Integrating Rankings into Quantized Scores in Peer Review [61.27794774537103]
  In peer review, reviewers are usually asked to provide scores for the papers. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are no standard procedure for using this ranking information and Area Chairs may use it in different ways. We take a principled approach to integrate the ranking information into the scores.
  arXiv  Detail & Related papers  (2022-04-05T19:39:13Z)
• Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL Rolling Review and Beyond [58.71736531356398]
  We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow. We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
  arXiv  Detail & Related papers  (2022-01-27T11:02:43Z)
• Link Prediction on N-ary Relational Data Based on Relatedness Evaluation [61.61555159755858]
  We propose a method called NaLP to conduct link prediction on n-ary relational data. We represent each n-ary relational fact as a set of its role and role-value pairs. Experimental results validate the effectiveness and merits of the proposed methods.
  arXiv  Detail & Related papers  (2021-04-21T09:06:54Z)
• Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis? [40.75458460149429]
  We discuss whether datasets and tasks should be deemed off-limits for NLP research. We focus in particular on the role of data statements in ethically assessing research. We examine the outcomes of similar debates in other scientific disciplines.
  arXiv  Detail & Related papers  (2020-05-27T07:31:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.