VERIRAG: Healthcare Claim Verification via Statistical Audit in Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2507.17948v1
• Date: Wed, 23 Jul 2025 21:32:50 GMT
• Title: VERIRAG: Healthcare Claim Verification via Statistical Audit in Retrieval-Augmented Generation
• Abstract summary: VERIRAG is a framework that makes three notable contributions: (i) the Veritable, an 11-point checklist that evaluates each source for methodological rigor, including data integrity and statistical validity; (ii) a Hard-to-Vary (HV) Score, a quantitative aggregator that weights evidence by its quality and diversity; and (iii) a Dynamic Acceptance Threshold, which calibrates the required evidence based on how extraordinary a claim is.
• Score: 12.545868971471844
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation (RAG) systems are increasingly adopted in clinical decision support, yet they remain methodologically blind-they retrieve evidence but cannot vet its scientific quality. A paper claiming "Antioxidant proteins decreased after alloferon treatment" and a rigorous multi-laboratory replication study will be treated as equally credible, even if the former lacked scientific rigor or was even retracted. To address this challenge, we introduce VERIRAG, a framework that makes three notable contributions: (i) the Veritable, an 11-point checklist that evaluates each source for methodological rigor, including data integrity and statistical validity; (ii) a Hard-to-Vary (HV) Score, a quantitative aggregator that weights evidence by its quality and diversity; and (iii) a Dynamic Acceptance Threshold, which calibrates the required evidence based on how extraordinary a claim is. Across four datasets-comprising retracted, conflicting, comprehensive, and settled science corpora-the VERIRAG approach consistently outperforms all baselines, achieving absolute F1 scores ranging from 0.53 to 0.65, representing a 10 to 14 point improvement over the next-best method in each respective dataset. We will release all materials necessary for reproducing our results.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• Benchmarking Machine Learning Models for Multi-Omics-Based Breast Cancer Prediction [0.0]
  Estrogen Receptor (ER) status is a critical biomarker in breast cancer diagnosis, prognosis, and treatment selection.<n>This study presents a systematic benchmarking analysis of classical machine learning models for ER status prediction.
  arXiv  Detail & Related papers  (2026-06-27T02:46:40Z)
• ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence [57.37494162084001]
  Chain-of-Evidence (CoE) is a verifiability framework requiring every claim to be traceable to its evidence source.<n>CoE Audit is a post-hoc audit whose four integrity checks -- score verification, specification violation, reference verification, and method-code alignment -- apply uniformly to all systems.
  arXiv  Detail & Related papers  (2026-05-25T21:30:27Z)
• Ensembles of Large Language Models for Identifying EQ-5D Studies in PubMed Based on Their Abstracts [0.30786914102688595]
  Classifying studies that clearly report health-related quality-of-life results, such as EQ-5D data, requires a high level of clinical interpretation.<n>This study investigates the use of Google's Gemini and Gemma large language models (LLMs) in automating EQ-5D detection in the PubMed biomedical database.
  arXiv  Detail & Related papers  (2026-04-24T21:42:39Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Eligibility-Aware Evidence Synthesis: An Agentic Framework for Clinical Trial Meta-Analysis [49.83108591873481]
  EligMeta is an agentic framework that integrates automated trial discovery with eligibility-aware meta-analysis.<n>The framework structures eligibility criteria and computes similarity-based study weights reflecting population alignment between target trials.
  arXiv  Detail & Related papers  (2026-04-03T03:18:50Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• From Evidence-Based Medicine to Knowledge Graph: Retrieval-Augmented Generation for Sports Rehabilitation and a Domain Benchmark [12.595335483488052]
  In medicine, large language models increasingly rely on retrieval-augmented generation (RAG) to ground outputs in up-to-date external evidence.<n>This study addresses two key gaps: (1) the lack of PICO alignment between queries and retrieved evidence, and (2) the absence of evidence hierarchy considerations during reranking.<n>We present a generalizable strategy for adapting EBM to graph-based RAG, integrating the PICO framework into knowledge graph construction and retrieval.
  arXiv  Detail & Related papers  (2026-01-01T05:20:54Z)
• Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights [42.23030568766672]
  Large language models (LLMs) are transforming the landscape of medicine.<n>Retrieval-augmented generation (RAG) has been widely adopted to address these limitations.<n>We present the most comprehensive expert evaluation of RAG in medicine to date.
  arXiv  Detail & Related papers  (2025-11-10T06:00:12Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Evaluating Large Language Models for Evidence-Based Clinical Question Answering [4.101088122511548]
  Large Language Models (LLMs) have demonstrated substantial progress in biomedical and clinical applications.<n>We curate a benchmark drawing from Cochrane systematic reviews and clinical guidelines.<n>We observe consistent performance patterns across sources and clinical domains.
  arXiv  Detail & Related papers  (2025-09-13T15:03:34Z)
• An Uncertainty-Aware Dynamic Decision Framework for Progressive Multi-Omics Integration in Classification Tasks [6.736267874971369]
  We propose an uncertainty-aware, multi-view dynamic decision framework for omics data classification.<n>We employ a fusion strategy based on Dempster-Shafer theory to integrate heterogeneous modalities.<n>In three datasets, over 50% of cases achieved accurate classification using a single omics modality.
  arXiv  Detail & Related papers  (2025-06-20T13:44:14Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n> PPCI estimates the treatment effect in a target experiment with unlabeled factual outcomes, retrievable zero-shot from a pre-trained model.<n>We validate our method on synthetic and real-world scientific data, offering solutions to instances not solvable by vanilla Empirical Risk Minimization.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• WithdrarXiv: A Large-Scale Dataset for Retraction Study [33.782357627001154]
  We present WithdrarXiv, the first large-scale dataset of withdrawn papers from arXiv.<n>We develop a comprehensive taxonomy of retraction reasons, identifying 10 distinct categories ranging from critical errors to policy violations.<n>We demonstrate a simple yet highly accurate zero-shot automatic categorization of retraction reasons, achieving a weighted average F1-score of 0.96.
  arXiv  Detail & Related papers  (2024-12-04T23:36:23Z)
• Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos [2.0735422289416605]
  Expert MES/UCEIS annotation is time-consuming and susceptible to inter-rater variability. CNN-based weakly-supervised models with end-to-end (e2e) training lack generalization to new disease scores. "Arges" is a deep learning framework that incorporates positional encoding to estimate disease severity scores in endoscopy.
  arXiv  Detail & Related papers  (2024-10-01T09:23:14Z)
• uMedSum: A Unified Framework for Advancing Medical Abstractive Summarization [23.173826980480936]
  Current methods often sacrifice key information for faithfulness or introduce confabulations when prioritizing informativeness. This paper presents a benchmark of six advanced abstractive summarization methods across three diverse datasets using five standardized metrics. We propose uMedSum, a modular hybrid summarization framework that introduces novel approaches for sequential confabulation removal followed by key missing information addition.
  arXiv  Detail & Related papers  (2024-08-22T03:08:49Z)
• Generating Scientific Claims for Zero-Shot Scientific Fact Checking [54.62086027306609]
  Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data. We propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences. We also demonstrate its usefulness in zero-shot fact checking for biomedical claims.
  arXiv  Detail & Related papers  (2022-03-24T11:29:20Z)
• Reinforcement Learning with Heterogeneous Data: Estimation and Inference [84.72174994749305]
  We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of a given policy, and the Auto-Clustered Policy Iteration (ACPI) for estimating the optimal policy in a given policy class. We present simulations to support our theoretical findings, and we conduct an empirical study on the standard MIMIC-III dataset.
  arXiv  Detail & Related papers  (2022-01-31T20:58:47Z)
• A robust kernel machine regression towards biomarker selection in multi-omics datasets of osteoporosis for drug discovery [2.2897244874280043]
  We propose "robust kernel machine regression (RobMR)," to improve the robustness of statistical machine regression and the diversity of fictional data. Experiments demonstrate that the proposed approach effectively identifies the inter-related risk factors of osteoporosis. The proposed approach can be applied be to any disease model multi-omics datasets are available.
  arXiv  Detail & Related papers  (2022-01-13T16:39:46Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)
• A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases [60.07352590494571]
  The aim of this study was to extend this approach to the observational setting using a standardized scalable framework. We demonstrate our framework by evaluating the effect of angiotensin-converting enzyme (ACE) inhibitors versus beta blockers on three efficacy and six safety outcomes.
  arXiv  Detail & Related papers  (2020-10-13T14:48:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.