SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

• URL: http://arxiv.org/abs/2606.30201v1
• Date: Mon, 29 Jun 2026 12:17:35 GMT
• Title: SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation
• Abstract summary: Current evaluation protocols for Vision-Language Models in Radiology Report Generation rely on report-level metrics that measure lexical overlap or aggregate clinical correctness.<n>We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG.
• Score: 1.5782980044380892
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical overlap or aggregate clinical correctness. However, such metrics do not test whether individual diagnostic statements stem from the actual pathological evidence visible in the image. This allows models to achieve competitive scores by exploiting learned priors or spurious correlations, a failure mode we refer to as vision shortcut. We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG. SHOVIR extends two spatially annotated chest X-ray datasets, MIMIC-CXR and PadChest-GR, with per-box CheXpert labels, and defines image-level and disease-level occlusion experiments that contrast baseline performance on clean images against localized, region-specific perturbations. Comparing predictions across these conditions isolates two failure modes at the disease-class level: direct shortcuts, where a finding persists after its visual evidence is removed, and contextual shortcuts, where detection degrades once co-occurring pathologies are occluded despite the target region remaining intact. Benchmarking eight state-of-the-art VLMs, we find that shortcut behavior varies substantially across architectures and datasets. Models achieving the highest baseline report quality do not necessarily rank highest in spatial grounding, revealing that clinically fluent generation can coexist with shallow reliance on visual evidence. These findings expose a blind spot in current RRG evaluation and motivate region-aware assessment protocols.

Related papers

• Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models [46.202482780666635]
  Graph-Hierarchicald Supervised Clinical Alignment reformulates image-report supervision as a hierarchical clinical alignment problem.<n>Our method structures this alignment as a disease-conditioned process, where supervision is decomposed into two levels.<n> Experiments on MIMIC-CXR, IU-Xray, and COV-CTR show that our method consistently improves performance on both conventional and clinical metrics.
  arXiv  Detail & Related papers  (2026-08-25T06:32:26Z)
• Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP [1.4336070582473752]
  We investigate how real-world shortcuts manifest across different layers of the medical CLIP-based model, MedCLIP.<n>We find that the final linear probes achieve a high AUROC but poor calibration in the models.<n>Patterns consistent with localised shortcuts, such as drains, appear at later layers, while patterns consistent with diffuse shortcuts, such as scanner-specific noise patterns, emerge earlier.
  arXiv  Detail & Related papers  (2026-08-12T14:08:52Z)
• PathReportEval: A Systematic Benchmark for Pathology Report Generation [5.733136272690028]
  We present a standardized benchmark and evaluation framework for pathology report generation.<n>The framework standardizes preprocessing, feature extraction, training, decoding, and evaluation.<n>A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness.
  arXiv  Detail & Related papers  (2026-07-20T18:59:47Z)
• Enhancing Brain MRI Anomaly Detection and Reasoning with ROI Rethink and Synthetic Data [9.98520042320808]
  We present BrReMark, a framework that introduces explicit region marking into brain MRI diagnosis.<n>The model first generates hypotheses about potential abnormalities and grounds them through explicit bounding box marking.<n>We integrate a domain randomization-based pathology synthesis augmentation strategy to improve the model's generalizability to out-of-distribution (OOD) data.
  arXiv  Detail & Related papers  (2026-06-24T14:41:27Z)
• CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays [13.566972599173084]
  CheXpercept is a sequential, multi-level perception benchmark for chest X-ray analysis.<n>It mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction.<n>CheXpercept contains 10,400 QA items derived from 2,100 CXRs, covering seven clinically critical pulmonary and cardiac lesions.
  arXiv  Detail & Related papers  (2026-06-19T01:10:24Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generation [15.331803613974365]
  We propose a novel dual-stage disease-aware framework for chest X-ray report generation.<n>In Stage1, our model learns Disease-Aware Semantic Tokens (DASTs) corresponding to specific pathology categories.<n>In Stage2, we introduce a Disease-Visual Attention Fusion module to integrate disease-aware representations with visual features.
  arXiv  Detail & Related papers  (2025-11-15T15:31:51Z)
• XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography [6.447908430647854]
  We present the first systematic benchmark for evaluating cross-modal interpretability in chest X-rays.<n>We generate visual explanations using cross-attention and similarity-based localization maps.<n>We quantitatively assess their alignment with radiologist-annotated regions across multiple pathologies.
  arXiv  Detail & Related papers  (2025-10-22T13:52:19Z)
• Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation [61.350584471060756]
  Vision-grounded medical report generation aims to produce clinically accurate descriptions of medical images.<n>We propose Self-Supervised Anatomical Consistency Learning (SS-ACL) to align generated reports with corresponding anatomical regions.<n>SS-ACL constructs a hierarchical anatomical graph inspired by the invariant top-down inclusion structure of human anatomy.
  arXiv  Detail & Related papers  (2025-09-30T08:59:06Z)
• You've Got Two Teachers: Co-evolutionary Image and Report Distillation for Semi-supervised Anatomical Abnormality Detection in Chest X-ray [33.272751884183556]
  Experimental results on the public MIMIC-CXR benchmark demonstrate CEIRD's superior performance to several up-to-date weakly and semi-supervised methods.
  arXiv  Detail & Related papers  (2023-07-18T12:18:21Z)
• Cross-Modal Causal Intervention for Medical Report Generation [107.76649943399168]
  Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance.<n> generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases.<n>We propose a two-stage framework named CrossModal Causal Representation Learning (CMCRL)<n> Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T07:23:55Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.