An Empirical Framework for Domain Generalization in Clinical Settings
- URL: http://arxiv.org/abs/2103.11163v1
- Date: Sat, 20 Mar 2021 11:48:57 GMT
- Title: An Empirical Framework for Domain Generalization in Clinical Settings
- Authors: Haoran Zhang, Natalie Dullerud, Laleh Seyyed-Kalantari, Quaid Morris,
Shalmali Joshi, Marzyeh Ghassemi
- Abstract summary: We benchmark the performance of eight domain generalization methods on multi-site clinical time series and medical imaging data.
We find that current domain generalization methods do not achieve significant gains in out-of-distribution performance over empirical risk minimization.
- Score: 14.363133217553715
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Clinical machine learning models experience significantly degraded
performance in datasets not seen during training, e.g., new hospitals or
populations. Recent developments in domain generalization offer a promising
solution to this problem by creating models that learn invariances across
environments. In this work, we benchmark the performance of eight domain
generalization methods on multi-site clinical time series and medical imaging
data. We introduce a framework to induce synthetic but realistic domain shifts
and sampling bias to stress-test these methods over existing non-healthcare
benchmarks. We find that current domain generalization methods do not achieve
significant gains in out-of-distribution performance over empirical risk
minimization on real-world medical imaging data, in line with prior work on
general imaging datasets. However, a subset of realistic induced-shift
scenarios in clinical time series data exhibit limited performance gains. We
characterize these scenarios in detail, and recommend best practices for domain
generalization in the clinical setting.
Related papers
- On the Out of Distribution Robustness of Foundation Models in Medical
Image Segmentation [47.95611203419802]
Foundations for vision and language, pre-trained on extensive sets of natural image and text data, have emerged as a promising approach.
We compare the generalization performance to unseen domains of various pre-trained models after being fine-tuned on the same in-distribution dataset.
We further developed a new Bayesian uncertainty estimation for frozen models and used them as an indicator to characterize the model's performance on out-of-distribution data.
arXiv Detail & Related papers (2023-11-18T14:52:10Z) - Generalization in medical AI: a perspective on developing scalable
models [3.003979691986621]
Many prestigious journals now require reporting results both on the local hidden test set as well as on external datasets.
This is because of the variability encountered in intended use and specificities across hospital cultures.
We establish a hierarchical three-level scale system reflecting the generalization level of a medical AI algorithm.
arXiv Detail & Related papers (2023-11-09T14:54:28Z) - Domain Generalization with Adversarial Intensity Attack for Medical
Image Segmentation [27.49427483473792]
In real-world scenarios, it is common for models to encounter data from new and different domains to which they were not exposed to during training.
domain generalization (DG) is a promising direction as it enables models to handle data from previously unseen domains.
We introduce a novel DG method called Adversarial Intensity Attack (AdverIN), which leverages adversarial training to generate training data with an infinite number of styles.
arXiv Detail & Related papers (2023-04-05T19:40:51Z) - Deployment of Image Analysis Algorithms under Prevalence Shifts [6.373765910269204]
Domain gaps are among the most relevant roadblocks in the clinical translation of machine learning (ML)-based solutions for medical image analysis.
We propose a workflow for prevalence-aware image classification that uses estimated deployment prevalences to adjust a trained classifier to a new environment.
arXiv Detail & Related papers (2023-03-22T13:16:37Z) - Rethinking Semi-Supervised Medical Image Segmentation: A
Variance-Reduction Perspective [51.70661197256033]
We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation.
We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks.
We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
arXiv Detail & Related papers (2023-02-03T13:50:25Z) - Single-domain Generalization in Medical Image Segmentation via Test-time
Adaptation from Shape Dictionary [64.5632303184502]
Domain generalization typically requires data from multiple source domains for model learning.
This paper studies the important yet challenging single domain generalization problem, in which a model is learned under the worst-case scenario with only one source domain to directly generalize to different unseen target domains.
We present a novel approach to address this problem in medical image segmentation, which extracts and integrates the semantic shape prior information of segmentation that are invariant across domains.
arXiv Detail & Related papers (2022-06-29T08:46:27Z) - Augmentation based unsupervised domain adaptation [2.304713283039168]
Deep learning models trained on small and unrepresentative data tend to outperform when deployed in data that differs from the one used for training.
Our approach takes advantage of the properties of adversarial domain adaptation and consistency training to achieve more robust adaptation.
arXiv Detail & Related papers (2022-02-23T13:06:07Z) - On the Practicality of Deterministic Epistemic Uncertainty [106.06571981780591]
deterministic uncertainty methods (DUMs) achieve strong performance on detecting out-of-distribution data.
It remains unclear whether DUMs are well calibrated and can seamlessly scale to real-world applications.
arXiv Detail & Related papers (2021-07-01T17:59:07Z) - Domain Generalization for Medical Imaging Classification with
Linear-Dependency Regularization [59.5104563755095]
We introduce a simple but effective approach to improve the generalization capability of deep neural networks in the field of medical imaging classification.
Motivated by the observation that the domain variability of the medical images is to some extent compact, we propose to learn a representative feature space through variational encoding.
arXiv Detail & Related papers (2020-09-27T12:30:30Z) - Shape-aware Meta-learning for Generalizing Prostate MRI Segmentation to
Unseen Domains [68.73614619875814]
We present a novel shape-aware meta-learning scheme to improve the model generalization in prostate MRI segmentation.
Experimental results show that our approach outperforms many state-of-the-art generalization methods consistently across all six settings of unseen domains.
arXiv Detail & Related papers (2020-07-04T07:56:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.