Robust Representation Learning with Self-Distillation for Domain Generalization

• URL: http://arxiv.org/abs/2302.06874v2
• Date: Fri, 12 Apr 2024 04:42:29 GMT
• Title: Robust Representation Learning with Self-Distillation for Domain Generalization
• Authors: Ankur Singh, Senthilnath Jayavelu,
• Abstract summary: We propose a novel domain generalization technique called Robust Representation Learning with Self-Distillation. We observe an average accuracy improvement in the range of 1.2% to 2.3% over the state-of-the-art on three datasets.
• Score: 2.0817769887373245
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the recent success of deep neural networks, there remains a need for effective methods to enhance domain generalization using vision transformers. In this paper, we propose a novel domain generalization technique called Robust Representation Learning with Self-Distillation (RRLD) comprising i) intermediate-block self-distillation and ii) augmentation-guided self-distillation to improve the generalization capabilities of transformer-based models on unseen domains. This approach enables the network to learn robust and general features that are invariant to different augmentations and domain shifts while effectively mitigating overfitting to source domains. To evaluate the effectiveness of our proposed method, we perform extensive experiments on PACS and OfficeHome benchmark datasets, as well as an industrial wafer semiconductor defect dataset. The results demonstrate that RRLD achieves robust and accurate generalization performance. We observe an average accuracy improvement in the range of 1.2% to 2.3% over the state-of-the-art on the three datasets.

Related papers

• Feature Fusion Transferability Aware Transformer for Unsupervised Domain Adaptation [1.9035011984138845]
  Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from labeled source domains to improve performance on unlabeled target domains. Recent research has shown promise in applying Vision Transformers (ViTs) to this task. We propose a novel Feature Fusion Transferability Aware Transformer (FFTAT) to enhance ViT performance in UDA tasks.
  arXiv  Detail & Related papers  (2024-11-10T22:23:12Z)
• Domain penalisation for improved Out-of-Distribution Generalisation [1.979158763744267]
  Domain generalisation (DG) aims to ensure robust performance across diverse and unseen target domains. We propose a framework for the task of object detection, where the data is assumed to be sampled from multiple source domains. By prioritising the domains that needs more attention, our approach effectively balances the training process.
  arXiv  Detail & Related papers  (2024-08-03T11:06:47Z)
• Self-Supervised Modality-Agnostic Pre-Training of Swin Transformers [0.7496510641958004]
  We augment the Swin Transformer to learn from different medical imaging modalities, enhancing downstream performance. Our model, dubbed SwinFUSE, offers three key advantages: (i) it learns from both Computed Tomography (CT) and Magnetic Resonance Images (MRI) during pre-training, resulting in complementary feature representations; (ii) a domain-invariance module (DIM) that effectively highlights salient input regions, enhancing adaptability; (iii) exhibits remarkable generalizability, surpassing the confines of tasks it was initially pre-trained on.
  arXiv  Detail & Related papers  (2024-05-21T13:28:32Z)
• CMDA: Cross-Modal and Domain Adversarial Adaptation for LiDAR-Based 3D Object Detection [14.063365469339812]
  LiDAR-based 3D Object Detection methods often do not generalize well to target domains outside the source (or training) data distribution. We introduce a novel unsupervised domain adaptation (UDA) method, called CMDA, which leverages visual semantic cues from an image modality. We also introduce a self-training-based learning strategy, wherein a model is adversarially trained to generate domain-invariant features.
  arXiv  Detail & Related papers  (2024-03-06T14:12:38Z)
• Improving Source-Free Target Adaptation with Vision Transformers Leveraging Domain Representation Images [8.626222763097335]
  Unsupervised Domain Adaptation (UDA) methods facilitate knowledge transfer from a labeled source domain to an unlabeled target domain. This paper presents an innovative method to bolster ViT performance in source-free target adaptation, beginning with an evaluation of how key, query, and value elements affect ViT outcomes. Domain Representation Images (DRIs) act as domain-specific markers, effortlessly merging with the training regimen.
  arXiv  Detail & Related papers  (2023-11-21T13:26:13Z)
• MADAv2: Advanced Multi-Anchor Based Active Domain Adaptation Segmentation [98.09845149258972]
  We introduce active sample selection to assist domain adaptation regarding the semantic segmentation task. With only a little workload to manually annotate these samples, the distortion of the target-domain distribution can be effectively alleviated. A powerful semi-supervised domain adaptation strategy is proposed to alleviate the long-tail distribution problem.
  arXiv  Detail & Related papers  (2023-01-18T07:55:22Z)
• Generalizability of Adversarial Robustness Under Distribution Shifts [57.767152566761304]
  We take a first step towards investigating the interplay between empirical and certified adversarial robustness on one hand and domain generalization on another. We train robust models on multiple domains and evaluate their accuracy and robustness on an unseen domain. We extend our study to cover a real-world medical application, in which adversarial augmentation significantly boosts the generalization of robustness with minimal effect on clean data accuracy.
  arXiv  Detail & Related papers  (2022-09-29T18:25:48Z)
• On Certifying and Improving Generalization to Unseen Domains [87.00662852876177]
  Domain Generalization aims to learn models whose performance remains high on unseen domains encountered at test-time. It is challenging to evaluate DG algorithms comprehensively using a few benchmark datasets. We propose a universal certification framework that can efficiently certify the worst-case performance of any DG method.
  arXiv  Detail & Related papers  (2022-06-24T16:29:43Z)
• Coarse to Fine: Domain Adaptive Crowd Counting via Adversarial Scoring Network [58.05473757538834]
  This paper proposes a novel adversarial scoring network (ASNet) to bridge the gap across domains from coarse to fine granularity. Three sets of migration experiments show that the proposed methods achieve state-of-the-art counting performance.
  arXiv  Detail & Related papers  (2021-07-27T14:47:24Z)
• Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency [90.71745178767203]
  Deep learning-based 3D object detection has achieved unprecedented success with the advent of large-scale autonomous driving datasets. Existing 3D domain adaptive detection methods often assume prior access to the target domain annotations, which is rarely feasible in the real world. We study a more realistic setting, unsupervised 3D domain adaptive detection, which only utilizes source domain annotations.
  arXiv  Detail & Related papers  (2021-07-23T17:19:23Z)
• Transformer-Based Source-Free Domain Adaptation [134.67078085569017]
  We study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. We propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA.
  arXiv  Detail & Related papers  (2021-05-28T23:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.