EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition

• URL: http://arxiv.org/abs/2304.01508v3
• Date: Tue, 27 Jun 2023 01:06:25 GMT
• Title: EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognition
• Authors: Siyuan Yan, Chi Liu, Zhen Yu, Lie Ju, Dwarikanath Mahapatrainst, Victoria Mar, Monika Janda, Peter Soyer, Zongyuan Ge
• Abstract summary: Skin lesion recognition using deep learning has made remarkable progress, and there is an increasing need for deploying these systems in real-world scenarios. Recent research has revealed that deep neural networks for skin lesion recognition may overly depend on disease-irrelevant image artifacts. We propose a novel domain generalization method called EPVT, which involves embedding prompts into the vision transformer to collaboratively learn knowledge from diverse domains.
• Score: 12.91556412209546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Skin lesion recognition using deep learning has made remarkable progress, and there is an increasing need for deploying these systems in real-world scenarios. However, recent research has revealed that deep neural networks for skin lesion recognition may overly depend on disease-irrelevant image artifacts (i.e., dark corners, dense hairs), leading to poor generalization in unseen environments. To address this issue, we propose a novel domain generalization method called EPVT, which involves embedding prompts into the vision transformer to collaboratively learn knowledge from diverse domains. Concretely, EPVT leverages a set of domain prompts, each of which plays as a domain expert, to capture domain-specific knowledge; and a shared prompt for general knowledge over the entire dataset. To facilitate knowledge sharing and the interaction of different prompts, we introduce a domain prompt generator that enables low-rank multiplicative updates between domain prompts and the shared prompt. A domain mixup strategy is additionally devised to reduce the co-occurring artifacts in each domain, which allows for more flexible decision margins and mitigates the issue of incorrectly assigned domain labels. Experiments on four out-of-distribution datasets and six different biased ISIC datasets demonstrate the superior generalization ability of EPVT in skin lesion recognition across various environments. Code is avaliable at https://github.com/SiyuanYan1/EPVT.

Related papers

• All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning [6.706482416007361]
  Domain generalization is critical in computational pathology (CPath)<n>We propose Domain Invariant Prompt Tuning (DIPT) for knowledge distillation process.<n>Our method adds a significant improvement in average F1-score to existing state-of-the-art knowledge distillation approaches.
  arXiv  Detail & Related papers  (2025-11-27T20:18:04Z)
• Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation [19.944946262284123]
  Humans can easily extrapolate novel domains, thus, an intriguing question arises: How can neural networks extrapolate like humans and achieve OOD generalization? We introduce a novel approach to domain extrapolation that leverages reasoning ability and the extensive knowledge encapsulated within large language models (LLMs) to synthesize entirely new domains. Our methods exhibit commendable performance in this setting, even surpassing the supervised setting by approximately 1-2% on datasets such as VLCS.
  arXiv  Detail & Related papers  (2024-03-08T18:44:23Z)
• Prompt-driven Latent Domain Generalization for Medical Image Classification [23.914889221925552]
  We propose a novel framework for medical image classification without relying on domain labels. PLDG consists of unsupervised domain discovery and prompt learning. Our method can achieve comparable or even superior performance than conventional DG algorithms.
  arXiv  Detail & Related papers  (2024-01-05T05:24:07Z)
• Single Domain Dynamic Generalization for Iris Presentation Attack Detection [41.126916126040655]
  Iris presentation generalization has achieved great success under intra-domain settings but easily degrades on unseen domains. We propose a Single Domain Dynamic Generalization (SDDG) framework, which exploits domain-invariant and domain-specific features on a per-sample basis. The proposed method is effective and outperforms the state-of-the-art on LivDet-Iris 2017 dataset.
  arXiv  Detail & Related papers  (2023-05-22T07:54:13Z)
• Context-aware Domain Adaptation for Time Series Anomaly Detection [69.3488037353497]
  Time series anomaly detection is a challenging task with a wide range of real-world applications. Recent efforts have been devoted to time series domain adaptation to leverage knowledge from similar domains. We propose a framework that combines context sampling and anomaly detection into a joint learning procedure.
  arXiv  Detail & Related papers  (2023-04-15T02:28:58Z)
• Towards Generalization on Real Domain for Single Image Dehazing via Meta-Learning [41.99615673136883]
  Internal information learned from synthesized images is usually sub-optimal in real domains. We present a domain generalization framework based on meta-learning to dig out representative internal properties of real hazy domains. Our proposed method has superior generalization ability than the state-of-the-art competitors.
  arXiv  Detail & Related papers  (2022-11-14T07:04:00Z)
• MD-CSDNetwork: Multi-Domain Cross Stitched Network for Deepfake Detection [80.83725644958633]
  Current deepfake generation methods leave discriminative artifacts in the frequency spectrum of fake images and videos. We present a novel approach, termed as MD-CSDNetwork, for combining the features in the spatial and frequency domains to mine a shared discriminative representation.
  arXiv  Detail & Related papers  (2021-09-15T14:11:53Z)
• Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting [75.80116276369694]
  In crowd counting, due to the problem of laborious labelling, it is perceived intractability of collecting a new large-scale dataset. We resort to the multi-domain joint learning and propose a simple but effective Domain-specific Knowledge Propagating Network (DKPNet) It is mainly achieved by proposing the novel Variational Attention(VA) technique for explicitly modeling the attention distributions for different domains.
  arXiv  Detail & Related papers  (2021-08-18T08:06:37Z)
• Open Domain Generalization with Domain-Augmented Meta-Learning [83.59952915761141]
  We study a novel and practical problem of Open Domain Generalization (OpenDG) We propose a Domain-Augmented Meta-Learning framework to learn open-domain generalizable representations. Experiment results on various multi-domain datasets demonstrate that the proposed Domain-Augmented Meta-Learning (DAML) outperforms prior methods for unseen domain recognition.
  arXiv  Detail & Related papers  (2021-04-08T09:12:24Z)
• DoFE: Domain-oriented Feature Embedding for Generalizable Fundus Image Segmentation on Unseen Datasets [96.92018649136217]
  We present a novel Domain-oriented Feature Embedding (DoFE) framework to improve the generalization ability of CNNs on unseen target domains. Our DoFE framework dynamically enriches the image features with additional domain prior knowledge learned from multi-source domains. Our framework generates satisfying segmentation results on unseen datasets and surpasses other domain generalization and network regularization methods.
  arXiv  Detail & Related papers  (2020-10-13T07:28:39Z)
• Cross-domain Face Presentation Attack Detection via Multi-domain Disentangled Representation Learning [109.42987031347582]
  Face presentation attack detection (PAD) has been an urgent problem to be solved in the face recognition systems. We propose an efficient disentangled representation learning for cross-domain face PAD. Our approach consists of disentangled representation learning (DR-Net) and multi-domain learning (MD-Net)
  arXiv  Detail & Related papers  (2020-04-04T15:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.