PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A

• URL: http://arxiv.org/abs/2602.21045v1
• Date: Tue, 24 Feb 2026 16:04:50 GMT
• Title: PaperTrail: A Claim-Evidence Interface for Grounding Provenance in LLM-based Scholarly Q&A
• Abstract summary: We introduce PaperTrail, a novel interface that decomposes large language models (LLMs) into discrete claims and evidence.<n>Our results show that PaperTrail significantly lowered participants' trust compared to the baseline interface.
• Score: 2.0532572974867174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used in scholarly question-answering (QA) systems to help researchers synthesize vast amounts of literature. However, these systems often produce subtle errors (e.g., unsupported claims, errors of omission), and current provenance mechanisms like source citations are not granular enough for the rigorous verification that scholarly domain requires. To address this, we introduce PaperTrail, a novel interface that decomposes both LLM answers and source documents into discrete claims and evidence, mapping them to reveal supported assertions, unsupported claims, and information omitted from the source texts. We evaluated PaperTrail in a within-subjects study with 26 researchers who performed two scholarly editing tasks using PaperTrail and a baseline interface. Our results show that PaperTrail significantly lowered participants' trust compared to the baseline. However, this increased caution did not translate to behavioral changes, as people continued to rely on LLM-generated scholarly edits to avoid a cognitively burdensome task. We discuss the value of claim-evidence matching for understanding LLM trustworthiness in scholarly settings, and present design implications for cognition-friendly communication of provenance information.

Related papers

• Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking [47.47518672198846]
  Misinformation spreading over the Internet poses a significant threat to both societies and individuals.<n>Previous methods rely on semantic and social-contextual patterns learned from training data.<n>We propose WKGFC, which exploits authorized open knowledge graph as a core resource of evidence.
  arXiv  Detail & Related papers  (2026-02-27T19:29:01Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading [24.52586571116556]
  Large Language Models (LLMs) increasingly serve as research assistants, yet their reliability in scholarly tasks remains under-evaluated.<n>In this work, we introduce PaperAsk, a benchmark that systematically evaluates LLMs across four key research tasks.<n>We find consistent reliability failures: citation retrieval fails in 48-98% of multi-reference queries, section-specific content extraction fails in 72-91% of cases, and topical paper discovery yields F1 scores below 0.32, missing over 60% of relevant literature.
  arXiv  Detail & Related papers  (2025-10-25T10:11:29Z)
• Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim $\rightarrow$ Evidence Reasoning [6.043212666944194]
  We present CLAIM-BENCH, a benchmark for evaluating large language models' capabilities in scientific claim-evidence extraction and validation.<n>We show that closed-source models like GPT-4 and Claude consistently outperform open-source counterparts in precision and recall.<n> strategically designed three-pass and one-by-one prompting approaches significantly improve LLMs' abilities to accurately link dispersed evidence with claims.
  arXiv  Detail & Related papers  (2025-06-09T21:04:39Z)
• The Feasibility of Topic-Based Watermarking on Academic Peer Reviews [46.71493672772134]
  We evaluate topic-based watermarking (TBW) for large language models (LLMs)<n>Our results show that TBW maintains review quality relative to non-watermarked outputs, while demonstrating strong to paraphrasing-based evasion.
  arXiv  Detail & Related papers  (2025-05-27T18:09:27Z)
• How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices? [1.130790932059036]
  We show that large language models (LLMs) reinforce the Matthew effect in citations by consistently favoring highly cited papers.<n>We analyze 274,951 references generated by GPT-4o for 10,000 papers.
  arXiv  Detail & Related papers  (2025-04-03T17:04:56Z)
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review.<n>The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system.<n>We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment [59.09144776166979]
  Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards.
  arXiv  Detail & Related papers  (2024-11-05T18:58:00Z)
• GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence [64.95492752484171]
  We present GenAudit -- a tool intended to assist fact-checking LLM responses for document-grounded tasks.<n>GenAudit suggests edits to the LLM response by revising or removing claims that are not supported by the reference document, and also presents evidence from the reference for facts that do appear to have support.<n> Comprehensive evaluation by human raters shows that GenAudit can detect errors in 8 different LLM outputs when summarizing documents from diverse domains.
  arXiv  Detail & Related papers  (2024-02-19T21:45:55Z)
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia [57.31074448586854]
  Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context. Yet the mechanisms underlying this contextual grounding remain unknown. We present a novel method to study grounding abilities using Fakepedia.
  arXiv  Detail & Related papers  (2023-12-04T17:35:42Z)
• When Large Language Models Meet Citation: A Survey [37.01594297337486]
  Large Language Models (LLMs) could be helpful in capturing fine-grained citation information via the corresponding textual context. Citations also establish connections among scientific papers, providing high-quality inter-document relationships. We review the application of LLMs for in-text citation analysis tasks, including citation classification, citation-based summarization, and citation recommendation.
  arXiv  Detail & Related papers  (2023-09-18T12:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.