A comprehensive multimodal dataset and benchmark for ulcerative colitis scoring in endoscopy

• URL: http://arxiv.org/abs/2603.14559v1
• Date: Sun, 15 Mar 2026 19:15:02 GMT
• Title: A comprehensive multimodal dataset and benchmark for ulcerative colitis scoring in endoscopy
• Abstract summary: Ulcerative colitis (UC) is a chronic mucosal inflammatory condition that places patients at increased risk of colorectal cancer.<n>The most widely used endoscopic scoring metrics are the Mayo Endoscopic Score (MES) and the Ulcerative Colitis Endoscopic Index of Severity (UCEIS)<n>We introduce a curated multi-centre, multi-resolution dataset that includes expert-validated MES and UCEIS labels, alongside detailed clinical descriptions.
• Score: 6.821894881874553
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Ulcerative colitis (UC) is a chronic mucosal inflammatory condition that places patients at increased risk of colorectal cancer. Colonoscopic surveillance remains the gold standard for assessing disease activity, and reporting typically relies on standardised endoscopic scoring metrics. The most widely used is the Mayo Endoscopic Score (MES), with some centres also adopting the Ulcerative Colitis Endoscopic Index of Severity (UCEIS). Both are descriptive assessments of mucosal inflammation (MES: 0 to 3; UCEIS: 0 to 8), where higher values indicate more severe disease. However, computational methods for automatically predicting these scores remain limited, largely due to the lack of publicly available expert-annotated datasets and the absence of robust benchmarking. There is also a significant research gap in generating clinically meaningful descriptions of UC images, despite image captioning being a well-established computer vision task. Variability in endoscopic systems and procedural workflows across centres further highlights the need for multi-centre datasets to ensure algorithmic robustness and generalisability. In this work, we introduce a curated multi-centre, multi-resolution dataset that includes expert-validated MES and UCEIS labels, alongside detailed clinical descriptions. To our knowledge, this is the first comprehensive dataset that combines dual scoring metrics for classification tasks with expert-generated captions describing mucosal appearance and clinically accepted reasoning for image captioning. This resource opens new opportunities for developing clinically meaningful multimodal algorithms. In addition to the dataset, we also provide benchmarking using convolutional neural networks, vision transformers, hybrid models, and widely used multimodal vision-language captioning algorithms.

Related papers

• MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry [32.93319761809706]
  Vision-Language Models (VLMs) have demonstrated significant potential in medical image analysis.<n>Their application in intraoral photography remains largely underexplored due to the lack of fine-grained, annotated datasets and comprehensive benchmarks.<n>We present MetaDent, a comprehensive resource that includes a novel and large-scale dentistry image dataset collected from clinical, public, and web sources.
  arXiv  Detail & Related papers  (2026-04-16T10:56:54Z)
• MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation [1.055294253870608]
  Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis.<n>Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings.<n>We study supervised adaptation of CLIP for MAD under a realistic clinical setting.
  arXiv  Detail & Related papers  (2026-03-18T03:37:44Z)
• Multimodal Carotid Risk Stratification with Large Vision-Language Models: Benchmarking, Fine-Tuning, and Clinical Insights [3.5469990240092373]
  This study investigates the potential of state-of-the-art and recent large vision-language models (LVLMs) for multimodal carotid plaque assessment.<n>A framework that simulates realistic diagnostic scenarios through interview-style question sequences is proposed.<n>Experiments reveal that even if LVLMs are very powerful, not all LVLMs can accurately identify imaging modality and anatomy.
  arXiv  Detail & Related papers  (2025-10-03T11:48:12Z)
• Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentation [16.773882069530426]
  We propose FOCUS-Med, which stands for Fusion of spatial and structural graph with attentional context-aware polyp segmentation.<n> FOCUS-Med integrates a Dual Graph Convolutional Network (Dual-GCN) module to capture contextual spatial and topological structural dependencies.<n>Experiments on public benchmarks demonstrate that FOCUS-Med achieves state-of-the-art performance across five key metrics.
  arXiv  Detail & Related papers  (2025-08-09T15:53:19Z)
• A Multimodal Approach For Endoscopic VCE Image Classification Using BiomedCLIP-PubMedBERT [0.62914438169038]
  This Paper presents an advanced approach for fine-tuning BiomedCLIP PubMedBERT, a multimodal model, to classify abnormalities in Video Capsule Endoscopy frames.<n>Our method categorizes images into ten specific classes: angioectasia, bleeding, erosion, erythema, foreign body, lymphangiectasia, polyp, ulcer, worms, and normal.<n>Performance metrics, including classification, accuracy, recall, and F1 score, indicate the models strong ability to accurately identify abnormalities in endoscopic frames.
  arXiv  Detail & Related papers  (2024-10-25T19:42:57Z)
• Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning [65.54680361074882]
  Eye-gaze Guided Multi-modal Alignment (EGMA) framework harnesses eye-gaze data for better alignment of medical visual and textual features. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets.
  arXiv  Detail & Related papers  (2024-03-19T03:59:14Z)
• Multi-task Explainable Skin Lesion Classification [54.76511683427566]
  We propose a few-shot-based approach for skin lesions that generalizes well with few labelled data. The proposed approach comprises a fusion of a segmentation network that acts as an attention module and classification network.
  arXiv  Detail & Related papers  (2023-10-11T05:49:47Z)
• GastroVision: A Multi-class Endoscopy Image Dataset for Computer Aided Gastrointestinal Disease Detection [6.231109933741383]
  This dataset includes different anatomical landmarks, pathological abnormalities, polyp removal cases and normal findings from the GI tract. It was annotated and verified by experienced GI endoscopists. We believe our dataset can facilitate the development of AI-based algorithms for GI disease detection and classification.
  arXiv  Detail & Related papers  (2023-07-16T19:36:03Z)
• Graph-Ensemble Learning Model for Multi-label Skin Lesion Classification using Dermoscopy and Clinical Images [7.159532626507458]
  This study introduces a Graph Convolution Network (GCN) to exploit prior co-occurrence between each category as a correlation matrix into the deep learning model for the multi-label classification. We propose a Graph-Ensemble Learning Model (GELN) that views the prediction from GCN as complementary information of the predictions from the fusion model.
  arXiv  Detail & Related papers  (2023-07-04T13:19:57Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
  We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG) CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure. Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
  arXiv  Detail & Related papers  (2022-06-04T13:16:30Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.