EyeBench: A Call for More Rigorous Evaluation of Retinal Image Enhancement

• URL: http://arxiv.org/abs/2502.14260v1
• Date: Thu, 20 Feb 2025 04:56:03 GMT
• Title: EyeBench: A Call for More Rigorous Evaluation of Retinal Image Enhancement
• Abstract summary: generative models have achieved significant success in enhancement fundus images.<n>The evaluation of these models still presents a considerable challenge.<n>We propose a novel comprehensive benchmark, EyeBench, to provide insights that align enhancement models with clinical needs.
• Score: 14.724629346280402
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Over the past decade, generative models have achieved significant success in enhancement fundus images.However, the evaluation of these models still presents a considerable challenge. A comprehensive evaluation benchmark for fundus image enhancement is indispensable for three main reasons: 1) The existing denoising metrics (e.g., PSNR, SSIM) are hardly to extend to downstream real-world clinical research (e.g., Vessel morphology consistency). 2) There is a lack of comprehensive evaluation for both paired and unpaired enhancement methods, along with the need for expert protocols to accurately assess clinical value. 3) An ideal evaluation system should provide insights to inform future developments of fundus image enhancement. To this end, we propose a novel comprehensive benchmark, EyeBench, to provide insights that align enhancement models with clinical needs, offering a foundation for future work to improve the clinical relevance and applicability of generative models for fundus image enhancement. EyeBench has three appealing properties: 1) multi-dimensional clinical alignment downstream evaluation: In addition to evaluating the enhancement task, we provide several clinically significant downstream tasks for fundus images, including vessel segmentation, DR grading, denoising generalization, and lesion segmentation. 2) Medical expert-guided evaluation design: We introduce a novel dataset that promote comprehensive and fair comparisons between paired and unpaired methods and includes a manual evaluation protocol by medical experts. 3) Valuable insights: Our benchmark study provides a comprehensive and rigorous evaluation of existing methods across different downstream tasks, assisting medical experts in making informed choices. Additionally, we offer further analysis of the challenges faced by existing methods. The code is available at \url{https://github.com/Retinal-Research/EyeBench}

Related papers

• ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding [67.57715989349471]
  Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge.<n>We introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding.<n>We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T17:59:49Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• Bridging Restoration and Diagnosis: A Comprehensive Benchmark for Retinal Fundus Enhancement [13.241000354696903]
  EyeBench-V2 is a benchmark designed to bridge the gap between enhancement model performance and clinical utility.<n>Our benchmark provides a rigorous, task-oriented analysis of existing generative models.
  arXiv  Detail & Related papers  (2026-04-04T17:24:23Z)
• Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study [6.850135803678454]
  We present a systematic evaluation of Segment Anything Model 3 (SAM3) for lesion segmentation.<n>Experiments on 13 datasets covering 11 lesion types demonstrate that SAM3 achieves strong cross-modality generalization.<n>These results highlight the potential of concept-based foundation models for scalable and practical medical image segmentation.
  arXiv  Detail & Related papers  (2026-03-26T22:30:10Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models [54.48710348910535]
  Existing medical reasoning benchmarks primarily focus on analyzing a patient's condition based on an image from a single visit.<n>We introduce TemMed-Bench, the first benchmark designed for analyzing changes in patients' conditions between different clinical visits.
  arXiv  Detail & Related papers  (2025-09-29T17:51:26Z)
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models [52.2001050216955]
  Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning.<n>We propose an expert-in-the-loop framework named Expert-Controlled-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training.
  arXiv  Detail & Related papers  (2025-07-12T09:03:30Z)
• Metrics that matter: Evaluating image quality metrics for medical image generation [48.85783422900129]
  This study comprehensively assesses commonly used no-reference image quality metrics using brain MRI data.<n>We evaluate metric sensitivity to a range of challenges, including noise, distribution shifts, and, critically, morphological alterations designed to mimic clinically relevant inaccuracies.
  arXiv  Detail & Related papers  (2025-05-12T01:57:25Z)
• The Skin Game: Revolutionizing Standards for AI Dermatology Model Comparison [0.6144680854063939]
  Deep Learning approaches in dermatological image classification have shown promising results, yet the field faces significant methodological challenges that impede proper evaluation.<n>This paper presents a systematic analysis of current methodological practices in skin disease classification research, revealing substantial inconsistencies in data preparation, augmentation strategies, and performance reporting.<n>We propose comprehensive methodological recommendations for model development, evaluation, and clinical deployment, emphasizing rigorous data preparation, systematic error analysis, and specialized protocols for different image types.
  arXiv  Detail & Related papers  (2025-02-04T17:15:36Z)
• Clinical Evaluation of Medical Image Synthesis: A Case Study in Wireless Capsule Endoscopy [63.39037092484374]
  This study focuses on the clinical evaluation of medical Synthetic Data Generation using Artificial Intelligence (AI) models. The paper contributes by a) presenting a protocol for the systematic evaluation of synthetic images by medical experts and b) applying it to assess TIDE-II, a novel variational autoencoder-based model for high-resolution WCE image synthesis. The results show that TIDE-II generates clinically relevant WCE images, helping to address data scarcity and enhance diagnostic tools.
  arXiv  Detail & Related papers  (2024-10-31T19:48:50Z)
• RIDGE: Reproducibility, Integrity, Dependability, Generalizability, and Efficiency Assessment of Medical Image Segmentation Models [1.4675465116143782]
  This paper introduces the RIDGE checklist to assess the Reproducibility, Integrity, Dependability, Generalizability, and Efficiency of deep learning-based medical image segmentation models. The RIDGE checklist is not just a tool for evaluation but also a guideline for researchers striving to improve the quality and transparency of their work.
  arXiv  Detail & Related papers  (2024-01-16T21:45:08Z)
• Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedback [22.390670838355295]
  This paper introduces a pathologist-in-the-loop framework for generating clinically-plausible synthetic medical images. We show that human feedback significantly improves the quality of synthetic images in terms of fidelity, diversity, utility in downstream applications, and plausibility as evaluated by experts.
  arXiv  Detail & Related papers  (2023-06-16T21:54:20Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• A Knowledge-based Learning Framework for Self-supervised Pre-training Towards Enhanced Recognition of Medical Images [14.304996977665212]
  This study proposes a knowledge-based learning framework towards enhanced recognition of medical images. It works in three phases by synergizing contrastive learning and generative learning models. The proposed framework statistically excels in self-supervised benchmarks, achieving 2.08, 1.23, 1.12, 0.76 and 1.38 percentage points improvements over SimCLR in AUC/Dice.
  arXiv  Detail & Related papers  (2022-11-27T03:58:58Z)
• Harmonizing Pathological and Normal Pixels for Pseudo-healthy Synthesis [68.5287824124996]
  We present a new type of discriminator, the segmentor, to accurately locate the lesions and improve the visual quality of pseudo-healthy images. We apply the generated images into medical image enhancement and utilize the enhanced results to cope with the low contrast problem. Comprehensive experiments on the T2 modality of BraTS demonstrate that the proposed method substantially outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T08:41:17Z)
• Towards a Guideline for Evaluation Metrics in Medical Image Segmentation [0.0]
  This work provides an overview and interpretation guide on the following metrics for medical image segmentation evaluation in binary. As a summary, we propose a guideline for standardized medical image segmentation evaluation.
  arXiv  Detail & Related papers  (2022-02-10T13:38:05Z)
• Deep AUC Maximization for Medical Image Classification: Challenges and Opportunities [60.079782224958414]
  We will present and discuss opportunities and challenges brought by a new deep learning method by AUC (aka underlinebf Deep underlinebf AUC classification)
  arXiv  Detail & Related papers  (2021-11-01T15:31:32Z)
• Cross-Site Severity Assessment of COVID-19 from CT Images via Domain Adaptation [64.59521853145368]
  Early and accurate severity assessment of Coronavirus disease 2019 (COVID-19) based on computed tomography (CT) images offers a great help to the estimation of intensive care unit event. To augment the labeled data and improve the generalization ability of the classification model, it is necessary to aggregate data from multiple sites. This task faces several challenges including class imbalance between mild and severe infections, domain distribution discrepancy between sites, and presence of heterogeneous features.
  arXiv  Detail & Related papers  (2021-09-08T07:56:51Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization [112.2628296775395]
  Clinical decision support using deep neural networks has become a topic of steadily growing interest. clinicians are often hesitant to adopt the technology because its underlying decision-making process is considered to be intransparent and difficult to comprehend. We propose a novel decision explanation scheme based on CycleGAN activation which generates high-quality visualizations of classifier decisions even in smaller data sets.
  arXiv  Detail & Related papers  (2020-10-09T14:39:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.