The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation

• URL: http://arxiv.org/abs/2603.28387v1
• Date: Mon, 30 Mar 2026 12:58:10 GMT
• Title: The Scaffold Effect: How Prompt Framing Drives Apparent Multimodal Gains in Clinical VLM Evaluation
• Abstract summary: We evaluate 12 open-weight vision-language models (VLMs) on binary classification across two clinical neuroimaging cohorts, textscFOR2107 (affective disorders) and textscOASIS-3 (cognitive decline)<n>Under these conditions, smaller VLMs exhibit gains of up to 58% F1 upon introduction of neuroimaging context, with distilled models becoming competitive with counterparts an order of magnitude larger.
• Score: 1.9655003184977389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Trustworthy clinical AI requires that performance gains reflect genuine evidence integration rather than surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on binary classification across two clinical neuroimaging cohorts, \textsc{FOR2107} (affective disorders) and \textsc{OASIS-3} (cognitive decline). Both datasets come with structural MRI data that carries no reliable individual-level diagnostic signal. Under these conditions, smaller VLMs exhibit gains of up to 58\% F1 upon introduction of neuroimaging context, with distilled models becoming competitive with counterparts an order of magnitude larger. A contrastive confidence analysis reveals that merely \emph{mentioning} MRI availability in the task prompt accounts for 70-80\% of this shift, independent of whether imaging data is present, a domain-specific instance of modality collapse we term the \emph{scaffold effect}. Expert evaluation reveals fabrication of neuroimaging-grounded justifications across all conditions, and preference alignment, while eliminating MRI-referencing behavior, collapses both conditions toward random baseline. Our findings demonstrate that surface evaluations are inadequate indicators of multimodal reasoning, with direct implications for the deployment of VLMs in clinical settings.

Related papers

• Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation [11.89113336167412]
  We benchmark state-of-the-art Vision-Language Models (VLMs) on lumbar spine MRI with a focus on diagnostic accuracy.<n>We propose an architecture-agnostic framework that augments VLM inputs with spatially localized, disc-level anomaly heatmaps.
  arXiv  Detail & Related papers  (2026-08-07T11:21:33Z)
• SIINR: Structurally Informed Implicit Neural Representations for super-resolution with uncertainty quantification of clinical quality diffusion MRI datasets [3.1420911290508897]
  We introduce SIINR, a framework for super-resoltion of clinical dMRI datasets.<n>We validate framework on a diverse set of open-access dMRI datasets.
  arXiv  Detail & Related papers  (2026-07-22T09:16:23Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Motion-Guided Causal Disentanglement for Robust Multi-View Cine Cardiac MRI Diagnosis [0.4638331367650009]
  We propose a Motion-Guided View-Disease Disentanglement framework MoViD built upon a ViT-MAE backbone.<n>We evaluate the framework on a private clinical venous thrombosis dataset and two public benchmarks.
  arXiv  Detail & Related papers  (2026-06-03T03:45:11Z)
• Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models [56.29123284262618]
  This work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty.<n>We analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class.<n>Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; and (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability
  arXiv  Detail & Related papers  (2026-04-13T03:59:54Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• Automated Lesion Segmentation of Stroke MRI Using nnU-Net: A Comprehensive External Validation Across Acute and Chronic Lesions [0.0]
  We evaluate stroke lesion segmentation using the nnU-Net framework across multiple publicly available MRI datasets.<n>Across stroke stages, models showed robust generalisation, with segmentation accuracy approaching reported inter-rater reliability.<n>In acute stroke, DWI-trained models consistently outperformed FLAIR-based models, with only modest gains from multimodal combinations.<n>In chronic stroke, increasing training set size improved performance, with diminishing returns beyond several hundred cases.
  arXiv  Detail & Related papers  (2026-01-13T16:29:20Z)
• Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models [43.46006663176283]
  This work systematically analyzes the impact of various perturbations on medical MLLMs.<n>For the visual modality, we propose a Perturbation-aware Denoising (PDC) which leverages MLLMs' own vision encoder to identify noise patterns.<n>For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text.
  arXiv  Detail & Related papers  (2025-12-26T10:23:30Z)
• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution [42.85462513661566]
  We present a lightweight, transparent multimodal ensemble that fuses physiological time-series measurements with unstructured clinical notes from the first 48 hours of an ICU stay.<n>A logistic regression model combines predictions from two modality-specific models: a bidirectional LSTM for vitals and a finetuned ClinicalModernBERT transformer for notes.<n>On the MIMIC-III benchmark, our late-fusion ensemble improves discrimination over the best single model while maintaining well-calibrated predictions.
  arXiv  Detail & Related papers  (2025-11-19T20:11:49Z)
• Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights [3.5469990240092373]
  This study investigates the potential of state-of-the-art and recent large vision-language models (LVLMs) for multimodal carotid plaque assessment.<n>A framework that simulates realistic diagnostic scenarios through interview-style question sequences is proposed.<n>Experiments reveal that even if LVLMs are very powerful, not all LVLMs can accurately identify imaging modality and anatomy.
  arXiv  Detail & Related papers  (2025-10-03T11:48:12Z)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks [21.203358914772465]
  Recent advances in vision-language models (VLMs) have achieved remarkable performance on standard medical benchmarks, yet their true clinical reasoning ability remains unclear.<n>We introduce Neural-MedBench, a compact yet reasoning-intensive benchmark designed to probe the limits of multimodal clinical reasoning in neurology.
  arXiv  Detail & Related papers  (2025-09-26T12:20:01Z)
• Benchmarking and Mitigate Sycophancy in Medical Vision-Language Models [21.353225217216252]
  Vision language models often exhibit sycophantic behavior prioritizing alignment with user phrasing social cues or perceived authority over evidence based reasoning.<n>This study evaluate clinical sycophancy in medical visual question answering through a novel clinically grounded benchmark.
  arXiv  Detail & Related papers  (2025-09-26T07:02:22Z)
• Metrics that matter: Evaluating image quality metrics for medical image generation [48.85783422900129]
  This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data.<n>We evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, morphological alterations designed to mimic clinically relevant inaccuracies.
  arXiv  Detail & Related papers  (2025-05-12T01:57:25Z)
• A Demographic-Conditioned Variational Autoencoder for fMRI Distribution Sampling and Removal of Confounds [49.34500499203579]
  We create a variational autoencoder (VAE)-based model, DemoVAE, to decorrelate fMRI features from demographics. We generate high-quality synthetic fMRI data based on user-supplied demographics.
  arXiv  Detail & Related papers  (2024-05-13T17:49:20Z)
• Assessment of Data Consistency through Cascades of Independently Recurrent Inference Machines for fast and robust accelerated MRI reconstruction [0.0]
  Data Consistency (DC) is crucial for generalization in multi-modal data and robustness in detecting pathology. This work proposes the Cascades of Independently Recurrent Inference Machines (CIRIM) to assess DC through unrolled optimization. We show that the CIRIM performs best when implicitly enforcing DC, while the E2EVN requires explicitly formulated DC.
  arXiv  Detail & Related papers  (2021-11-30T15:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.