RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing

• URL: http://arxiv.org/abs/2605.25956v1
• Date: Mon, 25 May 2026 15:30:02 GMT
• Title: RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing
• Abstract summary: We present RAPTOR+, a multimodal extension that uses Vision-Language Models (VLMs) for end-to-end referral understanding.<n>We evaluate fine-tuned VLMs, commercial and open-source zero-shot VLMs, and the original OCR-based pipeline on 223 clinically curated CRC urgent referral forms.
• Score: 1.384221236843682
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Urgent suspected colorectal cancer (CRC) referrals create operational bottlenecks because semi-structured clinical documents often require manual review and transcription. The original RAPTOR system used Large Language Models for structured extraction but relied on a separate OCR stage, making it vulnerable to handwriting, layout variation, and loss of visual evidence linkage. We present RAPTOR+, a multimodal extension that uses Vision-Language Models (VLMs) for end-to-end referral understanding. We evaluate fine-tuned VLMs, commercial and open-source zero-shot VLMs, and the original OCR-based pipeline on 223 clinically curated CRC urgent referral forms. We also introduce a grounding-aware evaluation framework that measures both extraction accuracy and evidence localisation. Results show a clear grounding gap in zero-shot models. Gemini 2.5 Flash achieved 92.6% Reading Accuracy but only 1.2% Strict Safety. In contrast, fine-tuned Qwen3-VL-8B achieved 96.1% Reading Accuracy and 60.6% Strict Safety, substantially improving verifiable evidence grounding. These findings show that task-specific fine-tuning is essential for reliable, auditable clinical document understanding. RAPTOR+ enables extracted referral decisions to be linked to visual evidence, supporting safer and more efficient cancer referral triage.

Related papers

• CARDEA: Auditable Reasoning Grounded in Spatial Evidence for End-to-End Coronary Angiography Interpretation [0.0]
  Existing AI systems can improve consistency but lack auditable decision processes.<n>We developed CARDEA, a unified large vision-language model that serves as the inference core of a CAG pipeline.<n> CARDEA runs an end-to-end CAG pipeline from raw multiview videos through selection to study-level diagnosis.
  arXiv  Detail & Related papers  (2026-09-07T02:01:27Z)
• A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports [58.14965296904734]
  EndoCLIP is a vision-language foundation model trained on 125,756 lesion-level image-text pairs.<n>It outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings.
  arXiv  Detail & Related papers  (2026-07-30T16:24:36Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• CuraView: A Multi-Agent Framework for Medical Hallucination Detection with GraphRAG-Enhanced Knowledge Verification [3.422186949568493]
  We present CuraView, a framework for sentence-level detection and evidence-grounded explanation of faithfulness hallucinations in discharge summaries.<n>CuraView constructs a GraphRAG-based knowledge graph from patient-level EHRs.<n>We evaluate CuraView on a subset of 250 patients from the Discharge-Me benchmark, with 50 patients held out for testing.
  arXiv  Detail & Related papers  (2026-05-05T08:05:31Z)
• Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models [53.279391576560755]
  Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across a wide range of NLP tasks, yet they remain prone to factual inaccuracies and limitation.<n>This poses significant risks in high-stakes domains such as healthcare, law, and scientific communication, where trust and verifiability are paramount.<n>We introduce DAVinCI - a Dual Attribution and Verification framework designed to enhance the factual reliability and interpretability of LLM outputs.
  arXiv  Detail & Related papers  (2026-04-23T01:37:50Z)
• Calibrated Confidence Expression for Radiology Report Generation [33.24673060327421]
  Large Vision-Language Models (LVLMs) in radiology report generation require accurate predictions and clinically interpretable indicators.<n>Current state-of-the-art language models are often overconfident, and research on calibration in multimodal settings such as radiology report generation is limited.<n>We introduce ConRad, a medical reinforcement learning framework for fine-tuning LVLMs to produce verbalized confidence estimates alongside radiology reports.
  arXiv  Detail & Related papers  (2026-03-31T09:37:33Z)
• EviSearch: A Human in the Loop System for Extracting and Auditing Clinical Evidence for Systematic Reviews [7.325762712182949]
  EviSearch is a multi-agent extraction system for clinical evidence tables.<n>The pipeline is designed for high-precision extraction across multimodal evidence sources.<n>EviSearch is intended to accelerate living systematic review, reduce manual curation burden, and provide a safe, auditable path for integrating LLM-based extraction into evidence synthesis pipelines.
  arXiv  Detail & Related papers  (2026-03-23T22:36:54Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• NAACL: Noise-AwAre Verbal Confidence Calibration for LLMs in RAG Systems [53.52419750390942]
  Large language models (LLMs) are used in mission-critical factual domains.<n>LLMs exhibit poor calibration performance due to noisy retrieved contexts.<n>We propose NAACL Rules (Noise-AwAre Confidence CaLibration Rules) to provide a principled foundation for resolving overconfidence under noise.
  arXiv  Detail & Related papers  (2026-01-16T05:38:25Z)
• CARE-RAG - Clinical Assessment and Reasoning in RAG [43.1450755645803]
  We study the gap between retrieval and reasoning in large language models (LLMs)<n>We propose an evaluation framework that measures accuracy, consistency, and fidelity of reasoning.
  arXiv  Detail & Related papers  (2025-11-20T02:44:55Z)
• Look As You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning [55.232400251303794]
  Look As You Think (LAT) is a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution.<n>LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5.
  arXiv  Detail & Related papers  (2025-11-15T02:50:23Z)
• Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift [12.373281238541296]
  Medical vision-language models (VLMs) offer promise for clinical decision support, yet their reliability under distribution shifts remains a major concern for safe deployment.<n>We propose DRiFt, a structured feature decoupling framework that explicitly separates clinically relevant signals from task-agnostic noise.<n>Our approach improves in-distribution performance by +11.4% Top-1 accuracy and +3.3% Macro-F1 over prior prompt-based methods.
  arXiv  Detail & Related papers  (2025-09-11T12:26:57Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.