CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
- URL: http://arxiv.org/abs/2605.12882v1
- Date: Wed, 13 May 2026 01:54:42 GMT
- Title: CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
- Abstract summary: We introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer.<n>CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document.<n>At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct.
- Score: 35.06938031991452
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answer and leave the supporting evidence unchecked. This answer-only approach masks a critical failure mode: a model can land on the correct answer while grounding it in the wrong passage -- a critical risk in high-stakes domains like law, finance, and medicine, where every conclusion must be traceable to a specific source region. To address this, we introduce CiteVQA, a benchmark that requires models to return element-level bounding-box citations alongside each answer, evaluating both jointly. CiteVQA comprises 1,897 questions across 711 PDFs spanning seven domains and two languages, averaging 40.6 pages per document. To ensure fidelity and scalability, the ground-truth citations are generated by an automated pipeline-which identifies crucial evidence via masking ablation-and are subsequently validated through expert review. At the core of our evaluation is Strict Attributed Accuracy (SAA), which credits a prediction only when the answer and the cited region are both correct. Auditing 20 MLLMs reveals a pervasive Attribution Hallucination: models frequently produce the right answer while citing the wrong region. The strongest system (Gemini-3.1-Pro-Preview) achieves an SAA of only 76.0, and the strongest open-source MLLM reaches just 22.5. Ultimately, towards trustworthy document intelligence, CiteVQA exposes a reliability gap that answer-only evaluations overlook, providing the instrumentation needed to close it. Our repository is available at https://github.com/opendatalab/CiteVQA.
Related papers
- NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
arXiv Detail & Related papers (2026-09-10T08:34:56Z) - AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents [21.842813905857863]
AtomCite parses an answer into claims, checks each claim against the image of the cited page, and applies a deterministic repair policy.<n>It reaches around 93% binary verification accuracy on the injected benchmark, significantly outperforming every OCR-only condition.<n>Its repair policy lifts citation precision on the injected mix from a constructed 34% to 87-90% while retaining over 90% of correct claims.
arXiv Detail & Related papers (2026-09-05T01:31:50Z) - ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers [0.0]
ResearchQA is a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers.<n>ResearchQA is designed for citation-grounded evaluation.<n>We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting.
arXiv Detail & Related papers (2026-07-13T04:26:51Z) - How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation [3.4562767039451674]
We show how a large language model decides whether to attach an inline citation while answering a factoid question.<n>We show that amplifying or attenuating only those critical heads and eliminating 69% of spurious ones on PopQA without harming answer accuracy.<n>The results reveal a potential disconnect between the model's apparent reasoning and its internal computational pathway.
arXiv Detail & Related papers (2026-06-09T07:06:13Z) - DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding [29.270128057512284]
We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem.<n>We design a four-stage evaluation protocol that audits each level of the trajectory independently.<n>We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems.
arXiv Detail & Related papers (2026-05-09T11:12:59Z) - Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents [0.18762753243053634]
Large language models (LLMs) power deep research agents that synthesize information from hundreds of web sources into cited reports.<n>Current approaches either trust models to self-cite accurately, risking bias, or employ retrieval-augmented generation (RAG) that does not validate source accessibility, relevance or factual consistency.<n>We introduce the first source attribution evaluation framework that uses a reproducible AST rubric to extract and evaluate inline citations at scale.
arXiv Detail & Related papers (2026-05-07T17:46:45Z) - HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
arXiv Detail & Related papers (2026-02-15T02:50:15Z) - VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
arXiv Detail & Related papers (2025-11-25T04:14:52Z) - CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
arXiv Detail & Related papers (2025-10-02T02:14:33Z) - Document Attribution: Examining Citation Relationships using Large Language Models [62.46146670035751]
We propose a zero-shot approach that frames attribution as a straightforward textual entailment task.<n>We also explore the role of the attention mechanism in enhancing the attribution process.
arXiv Detail & Related papers (2025-05-09T04:40:11Z) - Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation [8.975024781390077]
We present MIRAGE --Model Internals-based RAG Explanations -- a plug-and-play approach using model internals for faithful answer attribution in question answering applications.
We evaluate our proposed approach on a multilingual QA dataset, finding high agreement with human answer attribution.
arXiv Detail & Related papers (2024-06-19T16:10:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.