Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025

• URL: http://arxiv.org/abs/2608.09280v1
• Date: Mon, 10 Aug 2026 08:35:26 GMT
• Title: Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025
• Abstract summary: ACL released the EMNLP 2025 Checklists to aid transparency on the current research practice, which we focus on.<n>We provide the first analysis of recent EMNLP Checklists, by examining $73,922$ responses and justifications to them.<n>For the Main track, we find that authors isolate ethics questions of the Checklist from the paper's bulk, mimicking the trend of ethics being an afterthought.<n>We find significant issues with the checklist design and effort of authors, namely that $6%$ of all checklists contained logical contradictions between parent and child responses.
• Score: 3.5138213455573095
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Responsible NLP practice includes a) transparency, b) ethics, and c) societal impacts. The Responsible NLP Checklist aims to push these goals, and promote responsible practice. Recently, ACL released the EMNLP 2025 Checklists to aid transparency on the current research practice, which we focus on. We curate and release the first two datasets of: a) all the checklist responses and justifications from the EMNLP 2025 Main and Finding tracks; b) checklist reference linking to paper sections. We also provide the first analysis of recent EMNLP Checklists, by examining $73,922$ responses and justifications to them. For the Main track, we find that authors isolate ethics questions of the Checklist from the paper's bulk, mimicking the trend of ethics being an afterthought. We then examine \texttt{NO} responses. We find $44.9\%$ of justifications are poor or bad-faith, being brief or empty. Then, we find significant issues with the checklist design and effort of authors, namely that $6\%$ of all checklists contained logical contradictions between parent and child responses. We also find evidence of surface compliance for responsible ethics, with $53\%$ authors dismissing potential risks or social impacts of their work, for which there should be none. We compare this to the Findings track, noticing a similar trend in both tracks. Lastly, we discuss the implications of the checklist design and provide recommendations for future checklist iterations. Including: a) enforcing a minimum word count, b) enforcing more scrutiny on the risks of appliances.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• Commitment Checklist: Auditing Author Commitments in Peer Review [55.67642089544162]
  We present a large-scale audit of author commitments using large language models (LLMs) to compare rebuttals against camera-ready versions.<n>We find that while a majority of promised changes are implemented, a significant share (about 25%) are not, with "missing experiments" among the most frequently unfulfilled.<n>We propose the idea of Author Commitment Checklist, which would alert authors and organizers to unaddressed promises.
  arXiv  Detail & Related papers  (2026-01-08T05:37:09Z)
• Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment [59.09144776166979]
  Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards.
  arXiv  Detail & Related papers  (2024-11-05T18:58:00Z)
• ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses [5.170485483791494]
  ARR Responsible NLP Research checklist website states that the "checklist is designed to encourage best practices for responsible research"<n>We introduce ConfReady, a retrieval-augmented generation (RAG) application that can be used to empower authors to reflect on their work and assist authors with conference checklists.<n>To evaluate checklist assistants, we curate a dataset of 1,975 ACL checklist responses, analyze problems in human answers, and benchmark RAG and Large Language Model (LM) based systems on an evaluation subset.
  arXiv  Detail & Related papers  (2024-08-07T21:07:13Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)
• Alignment for Honesty [105.72465407518325]
  Recent research has made significant strides in aligning large language models (LLMs) with helpfulness and harmlessness. In this paper, we argue for the importance of alignment for emphhonesty, ensuring that LLMs proactively refuse to answer questions when they lack knowledge. We address these challenges by first establishing a precise problem definition and defining honesty'' inspired by the Analects of Confucius.
  arXiv  Detail & Related papers  (2023-12-12T06:10:42Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP [6.3596637237946725]
  This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research.
  arXiv  Detail & Related papers  (2021-09-14T11:36:42Z)
• Too Many Claims to Fact-Check: Prioritizing Political Claims Based on Check-Worthiness [1.2891210250935146]
  We propose a model prioritizing the claims based on their check-worthiness. We use BERT model with additional features including domain-specific controversial topics, word embeddings, and others.
  arXiv  Detail & Related papers  (2020-04-17T10:55:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.