NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

• URL: http://arxiv.org/abs/2603.24846v1
• Date: Wed, 25 Mar 2026 22:26:06 GMT
• Title: NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
• Abstract summary: We present a benchmarking study of vision-enabled large language models for 2D neuroimaging.<n>We use curated MRI and CT datasets covering multiple sclerosis, stroke, brain tumors, other abnormalities, and normal controls.<n> Tumor classification emerges as the most reliable task, stroke is moderately solvable, while multiple sclerosis and rare abnormalities remain difficult.
• Score: 0.46375328256275944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in multimodal large language models enable new possibilities for image-based decision support. However, their reliability and operational trade-offs in neuroimaging remain insufficiently understood. We present a comprehensive benchmarking study of vision-enabled large language models for 2D neuroimaging using curated MRI and CT datasets covering multiple sclerosis, stroke, brain tumors, other abnormalities, and normal controls. Models are required to generate multiple outputs simultaneously, including diagnosis, diagnosis subtype, imaging modality, specialized sequence, and anatomical plane. Performance is evaluated across four directions: discriminative classification with abstention, calibration, structured-output validity, and computational efficiency. A multi-phase framework ensures fair comparison while controlling for selection bias. Across twenty frontier multimodal models, the results show that technical imaging attributes such as modality and plane are nearly solved, whereas diagnostic reasoning, especially subtype prediction, remains challenging. Tumor classification emerges as the most reliable task, stroke is moderately solvable, while multiple sclerosis and rare abnormalities remain difficult. Few-shot prompting improves performance for several models but increases token usage, latency, and cost. Gemini-2.5-Pro and GPT-5-Chat achieve the strongest overall diagnostic performance, while Gemini-2.5-Flash offers the best efficiency-performance trade-off. Among open-weight architectures, MedGemma-1.5-4B demonstrates the most promising results, as under few-shot prompting, it approaches the zero-shot performance of several proprietary models, while maintaining perfect structured output. These findings provide practical insights into performance, reliability, and efficiency trade-offs, supporting standardized evaluation of multimodal LLMs in neuroimaging.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models [56.29123284262618]
  This work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty.<n>We analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class.<n>Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; and (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability
  arXiv  Detail & Related papers  (2026-04-13T03:59:54Z)
• Dual Distillation for Few-Shot Anomaly Detection [41.127862518102425]
  Anomaly detection is a critical task in computer vision with profound implications for medical imaging.<n>We introduce D$2$4FAD, a novel dual distillation framework for few-shot anomaly detection.<n>Our method identifies anomalies in previously unseen tasks using only a small number of normal reference images.
  arXiv  Detail & Related papers  (2026-03-02T10:38:19Z)
• A multimodal vision foundation model for generalizable knee pathology [40.03838145472935]
  Musculoskeletal disorders represent an urgent demand for precise interpretation of medical imaging.<n>Current artificial intelligence approaches in orthopedics rely on task-specific, supervised learning paradigms.<n>We introduce OrthoFoundation, a multimodal vision foundation model optimized for musculoskeletal pathology.
  arXiv  Detail & Related papers  (2026-01-26T08:14:51Z)
• MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysis [19.063517827476826]
  We introduce MM-DINOv2, a novel framework that adapts the pre-trained vision foundation model DINOv2 for multi-modal medical imaging.<n>Our approach incorporates multi-modal patch embeddings, enabling vision foundation models to effectively process multi-modal imaging data.<n>Our method achieves a Matthews Correlation Coefficient (MCC) of 0.6 on an external test set, surpassing state-of-the-art supervised approaches by +11.1%.
  arXiv  Detail & Related papers  (2025-09-08T12:34:15Z)
• Generalist versus Specialist Vision Foundation Models for Ocular Disease and Oculomics [10.703307148153433]
  We evaluate the adaptability of DINOv2 and DINOv3 in retinal image applications, compared to two specialist RETFound models.<n> RETFound-DINOv2 consistently outperforms these generalist foundation models in ocular-disease detection and oculomics tasks.
  arXiv  Detail & Related papers  (2025-09-03T15:48:57Z)
• Can Large Language Models Challenge CNNs in Medical Image Analysis? [3.1406146587437904]
  This study presents a multimodal AI framework designed for precisely classifying medical diagnostic images.<n>The proposed system compares the strengths of convolutional neural networks (CNNs) and different large language models (LLMs)
  arXiv  Detail & Related papers  (2025-05-29T14:48:09Z)
• Mind Your Vision: Multimodal Estimation of Refractive Disorders Using Electrooculography and Eye Tracking [12.016546264209536]
  This study explores a passive method for estimating refractive power using two eye movement recording techniques: electrooculography (EOG) and video-based eye tracking.<n>We trained Long Short-Term Memory (LSTM) models to classify refractive power from unimodal (EOG or eye tracking) and multimodal configuration.<n>Results show that the multimodal model consistently outperforms unimodal models, achieving the highest average accuracy in both settings.
  arXiv  Detail & Related papers  (2025-05-24T06:03:45Z)
• Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
  Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision. This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
  arXiv  Detail & Related papers  (2024-07-08T09:08:42Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis [58.634791552376235]
  Deep Learning (DL) models have achieved state-of-the-art performance in diagnosing multiple diseases using reconstructed images as input. DL models are sensitive to varying artifacts as it leads to changes in the input data distribution between the training and testing phases. We propose to use other normalization techniques, such as Group Normalization and Layer Normalization, to inject robustness into model performance against varying image artifacts.
  arXiv  Detail & Related papers  (2023-06-23T03:09:03Z)
• Detecting Dementia from Speech and Transcripts using Transformers [0.0]
  Alzheimer's disease (AD) constitutes a neurodegenerative disease with serious consequences to peoples' everyday lives, if it is not diagnosed early since there is no available cure. Current work has been focused on diagnosing dementia from spontaneous speech.
  arXiv  Detail & Related papers  (2021-10-27T21:00:01Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.