DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging

• URL: http://arxiv.org/abs/2602.02894v1
• Date: Mon, 02 Feb 2026 23:00:39 GMT
• Title: DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging
• Abstract summary: We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity.<n>We propose Counterfactual-Contrastive Inference, a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention.<n>On the MediConfusion benchmark, our approach achieves state-of-the-art performance, improving set-level accuracy by nearly 15% relative to prior methods while reducing confusion and improving individual accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate decision making in medical imaging requires reasoning over subtle visual differences between confusable conditions, yet most existing approaches rely on nearest neighbor retrieval that returns redundant evidence and reinforces a single hypothesis. We introduce a contrastive, document-aware reference selection framework that constructs compact evidence sets optimized for discrimination rather than similarity by explicitly balancing visual relevance, embedding diversity, and source-level provenance using ROCO embeddings and metadata. While ROCO provides large-scale image-caption pairs, it does not specify how references should be selected for contrastive reasoning, and naive retrieval frequently yields near-duplicate figures from the same document. To address this gap, we release a reproducible reference selection protocol and curated reference bank that enable a systematic study of contrastive retrieval in medical image reasoning. Building on these contrastive evidence sets, we propose Counterfactual-Contrastive Inference, a confidence-aware reasoning framework that performs structured pairwise visual comparisons and aggregates evidence using margin-based decision rules with faithful abstention. On the MediConfusion benchmark, our approach achieves state-of-the-art performance, improving set-level accuracy by nearly 15% relative to prior methods while reducing confusion and improving individual accuracy.

Related papers

• UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval [7.404190239665035]
  We propose a Unified Multimodal Embedding and Ranking framework for universal multimodal retrieval.<n>UMER replaces item-wise reflection with Pair-Aware Discriminative Reasoning.<n>On the MMEB-V2 benchmark, UMER achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-08-19T03:51:39Z)
• Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models [0.0]
  Medical vision-language models generate chain-of-thought reasoning before answering clinical questions.<n>We present CoT-Mediate, a behavioral framework that perturbs a single clinically meaningful attribute within a model's own generated reasoning.<n>We evaluate LLaVA-Med and MedGemma on 1,000 VQA-RAD samples each.
  arXiv  Detail & Related papers  (2026-07-29T17:11:52Z)
• Beyond the Leaderboard: Design Lessons for Trustworthy Multimodal VQA [4.080704298118247]
  multimodal AI must combine visual and textual evidence while remaining reliable and interpretable.<n>We analyze design choices across nine documented systems for question answering and explanation quality.<n>The findings support trustworthy multimodal healthcare AI based on data fusion, explainability, and resilient evaluation.
  arXiv  Detail & Related papers  (2026-07-16T17:38:19Z)
• Position: Anthropomorphic Misalignment Research Needs Stronger Evidence [52.76826423649968]
  We show how conceptual ambiguity, non-robust datasets, experimental design, and insufficient causal interventions can lead to overinterpretation of model behaviors.<n>This position paper aims to offer guidance on evidentiary considerations that can help improve methodological rigor in AMR.
  arXiv  Detail & Related papers  (2026-05-29T16:38:53Z)
• Case-Aware Medical Image Classification with Multimodal Knowledge Graphs and Reliability-Guided Refinement [27.443476988184994]
  We propose a case-aware reasoning framework driven by multimodal knowledge graphs for medical image classification.<n>Given an input image, our method adaptively retrieves similar cases from this memory and extracts their corresponding case-centered subgraphs.<n>Experiments on multiple medical imaging datasets demonstrate that our approach consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-05-21T14:29:20Z)
• Hit-RAG: Learning to Reason with Long Contexts via Preference Alignment [97.55382322103986]
  Hit-RAG is a multi-stage preference alignment framework designed to resolve cognitive bottlenecks.<n>Our approach systematically refines the utilization of external evidence via three distinct stages.
  arXiv  Detail & Related papers  (2026-03-07T04:05:27Z)
• BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation [33.53566598271416]
  BayesRAG is a novel multimodal retrieval framework grounded in Bayesian inference and Dempster-Shafer evidence theory.<n>We show that BayesRAG significantly outperforms state-of-the-art (SOTA) methods on challenging multimodal benchmarks.
  arXiv  Detail & Related papers  (2026-01-12T08:53:14Z)
• Factuality and Transparency Are All RAG Needs! Self-Explaining Contrastive Evidence Re-ranking [0.2864713389096699]
  This extended abstract introduces Self-Explaining Contrastive Evidence Re-Ranking (CER)<n>CER restructures retrieval around factual evidence by fine-tuning embeddings with contrastive learning and generating token-level attribution rationales for each retrieved passage.<n>We evaluated our method on clinical trial reports, and initial experimental results show that CER improves retrieval accuracy, mitigates the potential for hallucinations in RAG systems, and provides transparent, evidence-based retrieval that enhances reliability, especially in safety-critical domains.
  arXiv  Detail & Related papers  (2025-12-04T17:24:35Z)
• Exploring Health Misinformation Detection with Multi-Agent Debate [0.11470070927586014]
  We propose a two-stage framework for health misinformation detection.<n>In the first stage, we employ large language models (LLMs) to independently evaluate retrieved articles.<n>When this score indicates insufficient consensus-falling below a predefined threshold-the system proceeds to a second stage.<n>Multiple agents engage in structured debate to synthesize conflicting evidence and generate well-reasoned verdicts with explicit justifications.
  arXiv  Detail & Related papers  (2025-11-29T12:39:30Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrieval [12.135498957287004]
  We evaluate embeddings obtained from publicly available models and their domain-adapted variants.<n>We establish a systematic method to obtain thresholds for similarity scores for different embeddings.<n>We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones.
  arXiv  Detail & Related papers  (2024-06-18T07:03:34Z)
• SparseCL: Sparse Contrastive Learning for Contradiction Retrieval [87.02936971689817]
  Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query. Existing methods such as similarity search and crossencoder models exhibit significant limitations. We introduce SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences.
  arXiv  Detail & Related papers  (2024-06-15T21:57:03Z)
• ConPro: Learning Severity Representation for Medical Images using Contrastive Learning and Preference Optimization [30.31270613973337]
  This paper proposes Con- PrO: a novel representation learning method for severity assessment in medical images. We show that our representation learning framework offers valuable severity ordering in the feature space. We also derived discussions on severity indicators and related applications of preference comparison in the medical domain.
  arXiv  Detail & Related papers  (2024-04-29T16:16:42Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Fairness and robustness in anti-causal prediction [73.693135253335]
  Robustness to distribution shift and fairness have independently emerged as two important desiderata required of machine learning models. While these two desiderata seem related, the connection between them is often unclear in practice. By taking this perspective, we draw explicit connections between a common fairness criterion - separation - and a common notion of robustness.
  arXiv  Detail & Related papers  (2022-09-20T02:41:17Z)
• Robust Contrastive Learning against Noisy Views [79.71880076439297]
  We propose a new contrastive loss function that is robust against noisy views. We show that our approach provides consistent improvements over the state-of-the-art image, video, and graph contrastive learning benchmarks.
  arXiv  Detail & Related papers  (2022-01-12T05:24:29Z)
• Uncertainty-Aware Few-Shot Image Classification [118.72423376789062]
  Few-shot image classification learns to recognize new categories from limited labelled data. We propose Uncertainty-Aware Few-Shot framework for image classification.
  arXiv  Detail & Related papers  (2020-10-09T12:26:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.