LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

• URL: http://arxiv.org/abs/2608.07370v1
• Date: Fri, 07 Aug 2026 16:11:52 GMT
• Title: LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering
• Abstract summary: We present LitTraceQA, a benchmark for literature-grounded question answering over scientific papers.<n>Given a research question and a metadata pool of papers, a system must return three connected outputs.<n>We analyze a larger final annotation collection with 4,978 unique-question records over 4,859 unique gold papers.
• Score: 30.616598264585203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scientific literature is increasingly used as a knowledge source for language models, retrieval-augmented generation systems, and research assistants, but answering research questions from papers requires more than fluent generation. A reliable system must identify the relevant papers, locate the concrete evidence that supports the answer, and produce a response that is faithful to that evidence. We present LitTraceQA, a benchmark for literature-grounded question answering over scientific papers. Given a research question and a metadata pool of papers, a system must return three connected outputs: canonical paper identifiers, supporting evidence locations, and answers in one or more requested formats, including free-form text, multiple-choice answers, and structured tables. LitTraceQA targets evidence types common in scientific reading: tables, figures, text spans, equations or algorithms, and citation contexts. The public development split contains 55 examples, including 26 hidden-source single-paper questions and 29 multi-paper questions, and provides gold papers, evidence annotations, and answers for local validation. We also analyze a larger final annotation collection with 4,978 unique-question records over 4,859 unique gold papers. By evaluating paper retrieval, evidence grounding, and answer accuracy separately, LitTraceQA provides a testbed for scientific QA systems that produce verifiable answers rather than unsupported summaries.

Related papers

• Lit3R: Retrieve-Relate-Read for Evidence-Grounded Question Answering over Scientific Literature [4.514081720659125]
  We describe tus-nlp's Lit3R (Retrieve-Relate-Read) system for LitTraceQA.<n>Lit3R combines off-the-shelf retrieval, reranking, and large language model (LLM) components without task-specific training.<n>The reader first identifies supporting evidence within individual papers and then synthesizes evidence across papers to produce the final answer and evidence trace.
  arXiv  Detail & Related papers  (2026-09-15T09:50:15Z)
• Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation [60.151050016855834]
  DAS is a stateful agentic framework for generating publication-oriented academic surveys.<n>Its key idea is to separate reusable paper analysis from topic-specific manuscript construction.<n>DAS achieves the highest average in all four dimensions, with an overall score of 4.34 compared with 4.03 for the strongest competitor.
  arXiv  Detail & Related papers  (2026-08-18T17:29:00Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Explicit Evidence Grounding via Structured Inline Citation Generation [22.27784103991804]
  FullCite is a framework that generates structured inline citations linking each claim to both its source document and supporting evidence.<n>We assess citation quality and faithfulness along three dimensions: document-level correctness, evidence span identification, and claim-citation faithfulness.
  arXiv  Detail & Related papers  (2026-06-05T10:42:10Z)
• Adaptive Retrieval Strategies for Biomedical Question Answering [10.793379752589166]
  We propose an adaptive retrieval framework that selects retrieval and evidence aggregation strategies according to question type.<n>We evaluate the proposed framework on the BioASQ benchmark and demonstrate that adaptive retrieval strategies improve evidence relevance and answer quality.
  arXiv  Detail & Related papers  (2026-06-02T22:22:28Z)
• ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence [57.37494162084001]
  Chain-of-Evidence (CoE) is a verifiability framework requiring every claim to be traceable to its evidence source.<n>CoE Audit is a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems.
  arXiv  Detail & Related papers  (2026-05-25T21:30:27Z)
• SQuAI: Scientific Question-Answering with Multi-Agent Retrieval-Augmented Generation [4.224843546370802]
  SQuAI is a scalable and trustworthy multi-agent retrieval-augmented generation framework for scientific question answering.<n>Built on over 2.3 million full-text papers from arXiv.org, SQuAI employs four collaborative agents to decompose complex questions into sub-questions.<n>Our system improves faithfulness, answer relevance, and contextual relevance by up to +0.088 (12%) over a strong RAG baseline.
  arXiv  Detail & Related papers  (2025-10-17T14:20:55Z)
• VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification [107.75781898355562]
  We introduce a novel framework, called VeriCite, designed to rigorously validate supporting evidence and enhance answer attribution.<n>We conduct experiments across five open-source LLMs and four datasets, demonstrating that VeriCite can significantly improve citation quality while maintaining the correctness of the answers.
  arXiv  Detail & Related papers  (2025-10-13T13:38:54Z)
• PeerQA: A Scientific Question Answering Dataset from Peer Reviews [51.95579001315713]
  We present PeerQA, a real-world, scientific, document-level Question Answering dataset.<n>The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP.<n>We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks.
  arXiv  Detail & Related papers  (2025-02-19T12:24:46Z)
• SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers [20.273439120429025]
  SciDQA is a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles. Unlike other scientific QA datasets, SciDQA sources questions from peer reviews by domain experts and answers by paper authors. Questions in SciDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials.
  arXiv  Detail & Related papers  (2024-11-08T05:28:22Z)
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers [66.11048565324468]
  We present a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers.
  arXiv  Detail & Related papers  (2021-05-07T00:12:34Z)
• CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management [48.251211691263514]
  We present CAiRE-COVID, a real-time question answering (QA) and multi-document summarization system, which won one of the 10 tasks in the Kaggle COVID-19 Open Research dataset Challenge. Our system aims to tackle the recent challenge of mining the numerous scientific articles being published on COVID-19 by answering high priority questions from the community.
  arXiv  Detail & Related papers  (2020-05-04T15:07:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.