ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses

• URL: http://arxiv.org/abs/2408.04675v2
• Date: Fri, 19 Sep 2025 16:49:40 GMT
• Title: ConfReady: A RAG based Assistant and Dataset for Conference Checklist Responses
• Abstract summary: ARR Responsible NLP Research checklist website states that the "checklist is designed to encourage best practices for responsible research"<n>We introduce ConfReady, a retrieval-augmented generation (RAG) application that can be used to empower authors to reflect on their work and assist authors with conference checklists.<n>To evaluate checklist assistants, we curate a dataset of 1,975 ACL checklist responses, analyze problems in human answers, and benchmark RAG and Large Language Model (LM) based systems on an evaluation subset.
• Score: 5.170485483791494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ARR Responsible NLP Research checklist website states that the "checklist is designed to encourage best practices for responsible research, addressing issues of research ethics, societal impact and reproducibility." Answering the questions is an opportunity for authors to reflect on their work and make sure any shared scientific assets follow best practices. Ideally, considering a checklist before submission can favorably impact the writing of a research paper. However, previous research has shown that self-reported checklist responses don't always accurately represent papers. In this work, we introduce ConfReady, a retrieval-augmented generation (RAG) application that can be used to empower authors to reflect on their work and assist authors with conference checklists. To evaluate checklist assistants, we curate a dataset of 1,975 ACL checklist responses, analyze problems in human answers, and benchmark RAG and Large Language Model (LM) based systems on an evaluation subset. Our code is released under the AGPL-3.0 license on GitHub, with documentation covering the user interface and PyPI package.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025 [3.5138213455573095]
  ACL released the EMNLP 2025 Checklists to aid transparency on the current research practice, which we focus on.<n>We provide the first analysis of recent EMNLP Checklists, by examining $73,922$ responses and justifications to them.<n>For the Main track, we find that authors isolate ethics questions of the Checklist from the paper's bulk, mimicking the trend of ethics being an afterthought.<n>We find significant issues with the checklist design and effort of authors, namely that $6%$ of all checklists contained logical contradictions between parent and child responses.
  arXiv  Detail & Related papers  (2026-08-10T08:35:26Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• Author-in-the-Loop Response Generation and Evaluation: Integrating Author Expertise and Intent in Responses to Peer Review [53.99984738447279]
  Recent work frames this task as automatic text generation, underusing author expertise and intent.<n>We introduce REspGen, a generation framework that integrates explicit author input, multi-attribute control, and evaluation-guided refinement.<n>To support this formulation, we construct Re$3$Align, the first large-scale dataset of aligned review-response--revision triplets.
  arXiv  Detail & Related papers  (2026-01-19T14:07:10Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
  We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
  arXiv  Detail & Related papers  (2025-08-29T08:48:00Z)
• Checklists Are Better Than Reward Models For Aligning Language Models [99.1896531064102]
  We propose "Reinforcement Learning from Checklist Feedback" (RLCF)<n>From instructions, we extract checklists and evaluate how well responses satisfy each item.<n>Using both AI judges and specialized verifier programs, we combine these scores to compute rewards for RL.
  arXiv  Detail & Related papers  (2025-07-24T17:58:00Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• Read the Docs Before Rewriting: Equip Rewriter with Domain Knowledge via Continual Pre-training [35.17495480087131]
  A RAG-based question-answering system retrieves documents based on user queries.<n>In specialized domains, the rewriter model may struggle due to limited domain-specific knowledge.<n>We propose the R&R (Read the doc before Rewriting) rewriter, which involves continual pre-training on professional documents.
  arXiv  Detail & Related papers  (2025-07-01T06:51:00Z)
• From Prompting to Alignment: A Generative Framework for Query Recommendation [36.541332088115105]
  We propose a Generative Query Recommendation (GQR) framework that aligns query generation with user preference. Specifically, we unify diverse query recommendation tasks by a universal prompt framework. We also present a CTR-alignment framework, which involves training a query-wise CTR predictor as a process reward model.
  arXiv  Detail & Related papers  (2025-04-14T13:21:29Z)
• From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants [28.149173430599525]
  We use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources.<n>Our KG-RAG system retrieves relevant provenances, which are added to the user prompts context before being sent to the LLM generating the response.<n>Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.
  arXiv  Detail & Related papers  (2025-02-21T06:22:12Z)
• PeerQA: A Scientific Question Answering Dataset from Peer Reviews [51.95579001315713]
  We present PeerQA, a real-world, scientific, document-level Question Answering dataset.<n>The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP.<n>We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks.
  arXiv  Detail & Related papers  (2025-02-19T12:24:46Z)
• ACL-rlg: A Dataset for Reading List Generation [8.526112833986183]
  We introduce ACL-rlg, the largest open expert-annotated reading list dataset. Traditional scholarly search engines and indexing methods perform poorly on this task.
  arXiv  Detail & Related papers  (2024-12-30T07:48:32Z)
• Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment [59.09144776166979]
  Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards.
  arXiv  Detail & Related papers  (2024-11-05T18:58:00Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Exploring Requirements Elicitation from App Store User Reviews Using Large Language Models [0.0]
  This research introduces an approach leveraging the power of Large Language Models to analyze user reviews for automated requirements elicitation. We fine-tuned three well-established LLMs BERT, DistilBERT, and GEMMA, on a dataset of app reviews labeled for usefulness. Our evaluation revealed BERT's superior performance, achieving an accuracy of 92.40% and an F1-score of 92.39%, demonstrating its effectiveness in accurately classifying useful reviews.
  arXiv  Detail & Related papers  (2024-09-23T18:57:31Z)
• Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations [3.1911318265930944]
  ChatGPT uses third-party libraries nearly 10% more often than human developers. 14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT. We recommend that developers implement rigorous dependency management practices and double-check library licenses before integrating LLM-generated code into their projects.
  arXiv  Detail & Related papers  (2024-08-09T15:36:59Z)
• I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generation [60.00337758147594]
  This study explores the proactive ability of LLMs to seek user support. We propose metrics to evaluate the trade-off between performance improvements and user burden. Our experiments show that without external feedback, many LLMs struggle to recognize their need for user support.
  arXiv  Detail & Related papers  (2024-07-20T06:12:29Z)
• LitLLM: A Toolkit for Scientific Literature Review [15.785989492351684]
  We propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles. Our system first initiates a web search to retrieve relevant papers. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Third, the related work section is generated based on the re-ranked results and the abstract.
  arXiv  Detail & Related papers  (2024-02-02T02:41:28Z)
• The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions [114.67699010359637]
  We analyze a large-scale collection of real user queries to GPT. We find that tasks such as design'' and planning'' are prevalent in user interactions but are largely neglected or different from traditional NLP benchmarks.
  arXiv  Detail & Related papers  (2023-10-19T02:12:17Z)
• CRUISE-Screening: Living Literature Reviews Toolbox [8.292338880619061]
  CRUISE-Screening is a web-based application for conducting living literature reviews. It is connected to several search engines via an API, which allows for updating the search results periodically.
  arXiv  Detail & Related papers  (2023-09-04T15:58:43Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)
• Editable User Profiles for Controllable Text Recommendation [66.00743968792275]
  We propose LACE, a novel concept value bottleneck model for controllable text recommendations. LACE represents each user with a succinct set of human-readable concepts. It learns personalized representations of the concepts based on user documents.
  arXiv  Detail & Related papers  (2023-04-09T14:52:18Z)
• Recommendation Systems in Libraries: an Application with Heterogeneous Data Sources [66.81627042740679]
  The Reading&Machine project exploits the support of digitalization to increase the attractiveness of libraries and improve the users' experience. The project implements an application that helps the users in their decision-making process, providing recommendation system (RecSys)-generated lists of books the users might be interested in.
  arXiv  Detail & Related papers  (2023-03-21T11:13:01Z)
• Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP [6.3596637237946725]
  This position paper discusses the core legal and ethical principles for collection and sharing of textual data, and the tensions between them. We propose a potential checklist for responsible data (re-)use that could both standardise the peer review of conference submissions, as well as enable a more in-depth view of published research.
  arXiv  Detail & Related papers  (2021-09-14T11:36:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.