ACL-Verbatim: hallucination-free question answering for research

• URL: http://arxiv.org/abs/2605.21102v1
• Date: Wed, 20 May 2026 12:30:29 GMT
• Title: ACL-Verbatim: hallucination-free question answering for research
• Abstract summary: We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology.<n>We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers.<n>Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline.
• Score: 1.1852769462463206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Academic researchers need efficient and reliable methods for collecting high-quality information from trusted sources, but modern tools for AI-assisted research still suffer from the tendency of Large Language Models (LLMs) to produce factually inaccurate or nonsensical output, commonly referred to as hallucinations. We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology, directly mapping user queries to verbatim text spans in retrieved documents. We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers, and use it to train and evaluate a variety of extractive models. Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline based on the ScIRGen methodology, paired with chunks of research papers retrieved by VerbatimRAG. On this benchmark, a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7).

Related papers

• IntrAgent: An LLM Agent for Content-Grounded Information Retrieval through Literature Review [10.586701795115609]
  IntrAgent is designed to mimic human behaviors when reading literature for information retrieval.<n>It identifies relevant sections and then iteratively extracting key details to refine the retrieved information.<n>IntrAgent achieves 13.2% higher cross-domain accuracy than state-of-the-art RAG and research-agent baselines.
  arXiv  Detail & Related papers  (2026-04-23T01:55:08Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• SAGE: Benchmarking and Improving Retrieval for Deep Research Agents [60.53966065867568]
  We introduce SAGE, a benchmark for scientific literature retrieval comprising 1,200 queries across four scientific domains, with a 200,000 paper retrieval corpus.<n>We evaluate six deep research agents and find that all systems struggle with reasoning-intensive retrieval.<n> BM25 significantly outperforms LLM-based retrievers by approximately 30%, as existing agents generate keyword-oriented sub-queries.
  arXiv  Detail & Related papers  (2026-02-05T18:25:24Z)
• How Do LLM-Generated Texts Impact Term-Based Retrieval Models? [76.92519309816008]
  This paper investigates the influence of large language models (LLMs) on term-based retrieval models.<n>Our linguistic analysis reveals that LLM-generated texts exhibit smoother high-frequency and steeper low-frequency Zipf slopes.<n>Our study further explores whether term-based retrieval models demonstrate source bias, concluding that these models prioritize documents whose term distributions closely correspond to those of the queries.
  arXiv  Detail & Related papers  (2025-08-25T06:43:27Z)
• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs [48.73595915402094]
  MOLE is a framework that automatically extracts metadata attributes from scientific papers covering datasets of languages other than Arabic.<n>Our methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output.
  arXiv  Detail & Related papers  (2025-05-26T10:31:26Z)
• From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants [28.149173430599525]
  We use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources.<n>Our KG-RAG system retrieves relevant provenances, which are added to the user prompts context before being sent to the LLM generating the response.<n>Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.
  arXiv  Detail & Related papers  (2025-02-21T06:22:12Z)
• A Reproducibility and Generalizability Study of Large Language Models for Query Generation [14.172158182496295]
  generative AI and large language models (LLMs) promise to revolutionize the systematic literature review process. This paper presents an extensive study of Boolean query generation using LLMs for systematic reviews. Our study investigates the replicability and reliability of results achieved using ChatGPT. We then generalize our results by analyzing and evaluating open-source models.
  arXiv  Detail & Related papers  (2024-11-22T13:15:03Z)
• PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval [76.50690734636477]
  We propose PromptReps, which combines the advantages of both categories: no need for training and the ability to retrieve from the whole corpus. The retrieval system harnesses both dense text embedding and sparse bag-of-words representations.
  arXiv  Detail & Related papers  (2024-04-29T04:51:30Z)
• Cheap Ways of Extracting Clinical Markers from Texts [0.0]
  This paper describes the work of the UniBuc Archaeology team for CLPsych's 2024 Shared Task. It involved finding evidence within the text supporting the assigned suicide risk level. Two types of evidence were required: highlights and summaries.
  arXiv  Detail & Related papers  (2024-03-17T14:21:42Z)
• LitLLM: A Toolkit for Scientific Literature Review [15.785989492351684]
  We propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles.<n>Our system first initiates a web search to retrieve relevant papers.<n>Second, the system re-ranks the retrieved papers based on the user-provided abstract.<n>Third, the related work section is generated based on the re-ranked results and the abstract.
  arXiv  Detail & Related papers  (2024-02-02T02:41:28Z)
• Description-Based Text Similarity [59.552704474862004]
  We identify the need to search for texts based on abstract descriptions of their content. We propose an alternative model that significantly improves when used in standard nearest neighbor search.
  arXiv  Detail & Related papers  (2023-05-21T17:14:31Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.