Performance of large language models in the optical diagnosis of colorectal polyps

• URL: http://arxiv.org/abs/2608.07543v1
• Date: Fri, 31 Jul 2026 02:34:42 GMT
• Title: Performance of large language models in the optical diagnosis of colorectal polyps
• Abstract summary: Accurate optical diagnosis of colorectal polyps guides resection strategy and surveillance.<n> multimodal large language models (MLLMs) showing potential for image-based diagnosis.<n>We evaluated Claude Opus 4, Google Gemini 2.5 Pro, GPT-o3, GPT-4o, and GPT-5.
• Score: 26.586274034380523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Background and Study Aims: Accurate optical diagnosis of colorectal polyps guides resection strategy and surveillance, with multimodal large language models (MLLMs) showing potential for image-based diagnosis. We aimed to evaluate the diagnostic accuracy of MLLMs in classifying colorectal polyps and predicting histology. Methods: We conducted a retrospective diagnostic performance study using the PRIME dataset, a curated set of white light and narrow-band imaging (NBI) images. We evaluated Claude Opus 4, Google Gemini 2.5 Pro, GPT-o3, GPT-4o, and GPT-5. For Paris, Narrow-band Imaging Colorectal Endoscopic (NICE), and predicted histology, we calculated F1 scores, percent correct scores, and accuracy of each MLLM compared to expert responses for 132 cases. Cochran's Q and McNemar's Test were used to determine differences between predicted values of each MLLM. Results: The F1 scores among MLLMs were >0.9 for all models for neoplastic vs. non-neoplastic polyps. Gemini 2.5 Pro demonstrated the highest F1 scores for invasive vs. non-invasive polyps and low- vs. high-grade adenoma, at 0.560 and 0.492 respectively. Claude Opus 4 and GPT-5 had statistically significantly higher percent correct scores than other MLLMs at 41.7%, using Paris classification. Conclusions: Claude Opus 4 and Gemini 2.5 Pro showed the highest accuracy in differentiating polyp subtypes, performing closest to expert consensus. Sensitivity and specificity, however, did not meet ESGE standards, highlighting the need for prospective multicenter trials and the design of human-in-the-loop workflows before clinical deployment.

Related papers

• Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary [36.736436091313585]
  This commentary is the first controlled, cross-sectional evaluation of the GPT-5 family (GPT-5, GPT-5 Mini, GPT-5 Nano) against its predecessor GPT-4o.<n> GPT-5 demonstrated substantial gains in expert-level textual reasoning, with absolute improvements exceeding 25 percentage-points on MedXpertQA.<n>When tasked with multimodal synthesis, GPT-5 effectively leveraged this enhanced reasoning capacity to ground uncertain clinical narratives in concrete imaging evidence.
  arXiv  Detail & Related papers  (2026-03-05T03:24:48Z)
• GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards [17.453089229230663]
  We evaluate state-of-the-art Multimodal Large Language Models (MLLMs) across a panoramic gastrointestinal endoscopy workflow.<n>We constructed GI-Bench, a benchmark encompassing 20 fine-grained lesion categories.<n>Models generated reports with superior linguistic readability compared with humans.
  arXiv  Detail & Related papers  (2026-01-13T03:23:11Z)
• SkinGenBench: Generative Model and Preprocessing Effects for Synthetic Dermoscopic Augmentation in Melanoma Diagnosis [0.0]
  generative architecture choice has a stronger influence on both image fidelity and diagnostic utility than preprocessing complexity.<n>Synthetic data augmentation substantially improved melanoma detection with 8-15% absolute gains in melanoma F1-score.
  arXiv  Detail & Related papers  (2025-12-19T13:52:11Z)
• MeCaMIL: Causality-Aware Multiple Instance Learning for Fair and Interpretable Whole Slide Image Diagnosis [40.3028468133626]
  Multiple instance learning (MIL) has emerged as the dominant paradigm for whole slide image (WSI) analysis in computational pathology.<n>textbfMeCaMIL, a causality-aware MIL framework, explicitly models demographic confounders through structured causal graphs.<n>MeCaMIL achieves superior fairness -- demographic disparity variance drops by over 65% relative reduction on average across attributes.
  arXiv  Detail & Related papers  (2025-11-14T06:47:21Z)
• LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology [43.092364533480456]
  Vision-threatening eye diseases pose a major global health burden, with timely diagnosis limited by workforce shortages and restricted access to specialized care.<n>We present a large-scale multimodal ophthalmology benchmark comprising 32,633 instances with multi-granular annotations across 12 common ophthalmic conditions and 5 imaging modalities.<n>The dataset integrates imaging, anatomical structures, demographics, and free-text annotations, supporting anatomical structure recognition, disease screening, disease staging, and demographic prediction for bias evaluation.
  arXiv  Detail & Related papers  (2025-09-30T00:29:18Z)
• EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery [42.23133882924834]
  We develop textbfEyePCR, a large-scale benchmark for ophthalmic surgery analysis, grounded in structured clinical knowledge.<n>EyePCR offers a richly annotated corpus with more than 210k VQAs, which cover 1048 fine-grained attributes for multi-view perception.<n>Rich annotations facilitate in-depth cognitive analysis, simulating how surgeons perceive visual cues and combine them with domain knowledge to make decisions.
  arXiv  Detail & Related papers  (2025-09-19T04:55:56Z)
• Diagnostic Accuracy of Open-Source Vision-Language Models on Diverse Medical Imaging Tasks [1.6567957832859204]
  This dataset includes 22,349 images from 7,461 patients encompassing chest radiography, colon pathology, endoscopy, neonatal jaundice assessment, and retinal fundoscopy.<n>Qwen2.5 achieved the highest accuracy for chest radiographs (90.4%) and endoscopy images (84.2%), significantly outperforming the other models (p.001).<n>All models struggled with retinal fundoscopy; Qwen2.5 and Gemma3 achieved the highest, albeit modest, accuracies at 18.6% (comparable, p=.99), significantly better than other tested models (p.001)
  arXiv  Detail & Related papers  (2025-08-01T18:28:37Z)
• Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening [37.69303106863453]
  Large language models (LLMs) can simulate clinical reasoning based on natural language prompts, but their utility in ophthalmology is largely unexplored.<n>This study evaluated GPT-4's ability to interpret structured textual descriptions of retinal fundus photographs.<n>We conducted a retrospective diagnostic validation study using 300 annotated fundus images.
  arXiv  Detail & Related papers  (2025-07-02T01:35:59Z)
• Vision Language Models versus Machine Learning Models Performance on Polyp Detection and Classification in Colonoscopy Images [0.06782770175649853]
  This study provides a comprehensive performance assessment of vision-language models (VLMs) against established convolutional neural networks (CNNs)<n>We analyzed 2,258 colonoscopy images with corresponding pathology reports from 428 patients.
  arXiv  Detail & Related papers  (2025-03-27T09:41:35Z)
• Pre-examinations Improve Automated Metastases Detection on Cranial MRI [36.39673740985943]
  Automated MM detection on contrast-enhanced T1-weighted images performed with high sensitivity. Highest diagnostic performance was achieved by inclusion of only the contrast-enhanced T1-weighted images of the diagnosis and of a prediagnosis MRI.
  arXiv  Detail & Related papers  (2024-03-13T06:18:08Z)
• Domain Transfer Through Image-to-Image Translation for Uncertainty-Aware Prostate Cancer Classification [42.75911994044675]
  We present a novel approach for unpaired image-to-image translation of prostate MRIs and an uncertainty-aware training approach for classifying clinically significant PCa. Our approach involves a novel pipeline for translating unpaired 3.0T multi-parametric prostate MRIs to 1.5T, thereby augmenting the available training data. Our experiments demonstrate that the proposed method significantly improves the Area Under ROC Curve (AUC) by over 20% compared to the previous work.
  arXiv  Detail & Related papers  (2023-07-02T05:26:54Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.