FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models

• URL: http://arxiv.org/abs/2407.00983v2
• Date: Wed, 3 Jul 2024 16:37:36 GMT
• Title: FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models
• Authors: Ruinan Jin, Zikang Xu, Yuan Zhong, Qiongsong Yao, Qi Dou, S. Kevin Zhou, Xiaoxiao Li,
• Abstract summary: We introduce FairMedFM, a fairness benchmark for foundation models (FMs) research in medical imaging. FairMedFM integrates with 17 popular medical imaging datasets, encompassing different modalities, dimensionalities, and sensitive attributes. It explores 20 widely used FMs, with various usages such as zero-shot learning, linear probing, parameter-efficient fine-tuning, and prompting in various downstream tasks -- classification and segmentation.
• Score: 37.803490266325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The advent of foundation models (FMs) in healthcare offers unprecedented opportunities to enhance medical diagnostics through automated classification and segmentation tasks. However, these models also raise significant concerns about their fairness, especially when applied to diverse and underrepresented populations in healthcare applications. Currently, there is a lack of comprehensive benchmarks, standardized pipelines, and easily adaptable libraries to evaluate and understand the fairness performance of FMs in medical imaging, leading to considerable challenges in formulating and implementing solutions that ensure equitable outcomes across diverse patient populations. To fill this gap, we introduce FairMedFM, a fairness benchmark for FM research in medical imaging.FairMedFM integrates with 17 popular medical imaging datasets, encompassing different modalities, dimensionalities, and sensitive attributes. It explores 20 widely used FMs, with various usages such as zero-shot learning, linear probing, parameter-efficient fine-tuning, and prompting in various downstream tasks -- classification and segmentation. Our exhaustive analysis evaluates the fairness performance over different evaluation metrics from multiple perspectives, revealing the existence of bias, varied utility-fairness trade-offs on different FMs, consistent disparities on the same datasets regardless FMs, and limited effectiveness of existing unfairness mitigation methods. Checkout FairMedFM's project page and open-sourced codebase, which supports extendible functionalities and applications as well as inclusive for studies on FMs in medical imaging over the long term.

Related papers

• Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering [70.44269982045415]
  Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents.
  arXiv  Detail & Related papers  (2024-11-14T06:19:18Z)
• FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks [11.094602017349928]
  We propose FMBench, the first benchmark designed to evaluate the fairness of MLLMs performance across diverse demographic attributes. We thoroughly evaluate the performance and fairness of eight state-of-the-art open-source MLLMs, including both general and medical. All data and code will be released upon acceptance.
  arXiv  Detail & Related papers  (2024-10-01T21:38:15Z)
• Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation? [10.20366295974822]
  We introduce a novel decode head architecture, HQHSAM, which simply integrates elements from two state-of-the-art decoder heads, HSAM and HQSAM, to enhance segmentation performance. Our experiments on multiple datasets, encompassing various anatomies and modalities, reveal that FMs, particularly with the HQHSAM decode head, improve domain generalization for medical image segmentation.
  arXiv  Detail & Related papers  (2024-09-12T11:41:35Z)
• FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging [68.6715007665896]
  FedMedICL is a unified framework and benchmark to holistically evaluate federated medical imaging challenges. We comprehensively evaluate several popular methods on six diverse medical imaging datasets. We find that a simple batch balancing technique surpasses advanced methods in average performance across FedMedICL experiments.
  arXiv  Detail & Related papers  (2024-07-11T19:12:23Z)
• FeaInfNet: Diagnosis in Medical Image with Feature-Driven Inference and Visual Explanations [4.022446255159328]
  Interpretable deep learning models have received widespread attention in the field of image recognition. Many interpretability models that have been proposed still have problems of insufficient accuracy and interpretability in medical image disease diagnosis. We propose feature-driven inference network (FeaInfNet) to solve these problems.
  arXiv  Detail & Related papers  (2023-12-04T13:09:00Z)
• Ambiguous Medical Image Segmentation using Diffusion Models [60.378180265885945]
  We introduce a single diffusion model-based approach that produces multiple plausible outputs by learning a distribution over group insights. Our proposed model generates a distribution of segmentation masks by leveraging the inherent sampling process of diffusion. Comprehensive results show that our proposed approach outperforms existing state-of-the-art ambiguous segmentation networks.
  arXiv  Detail & Related papers  (2023-04-10T17:58:22Z)
• MEDFAIR: Benchmarking Fairness for Medical Imaging [44.73351338165214]
  MEDFAIR is a framework to benchmark the fairness of machine learning models for medical imaging. We find that the under-studied issue of model selection criterion can have a significant impact on fairness outcomes. We make recommendations for different medical application scenarios that require different ethical principles.
  arXiv  Detail & Related papers  (2022-10-04T16:30:47Z)
• Cross-Modal Information Maximization for Medical Imaging: CMIM [62.28852442561818]
  In hospitals, data are siloed to specific information systems that make the same information available under different modalities. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. We propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time.
  arXiv  Detail & Related papers  (2020-10-20T20:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.