Rethinking Domain Adaptation and Generalization in the Era of CLIP

• URL: http://arxiv.org/abs/2407.15173v1
• Date: Sun, 21 Jul 2024 14:09:14 GMT
• Title: Rethinking Domain Adaptation and Generalization in the Era of CLIP
• Authors: Ruoyu Feng, Tao Yu, Xin Jin, Xiaoyuan Yu, Lei Xiao, Zhibo Chen,
• Abstract summary: We show that a simple domain prior boosts CLIP's zero-shot recognition in a specific domain. We also create a benchmark for zero-shot adaptation and pseudo-labeling based self-training with CLIP. We propose to improve the task generalization ability of CLIP from multiple unlabeled domains.
• Score: 27.12334798260904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent studies on domain adaptation, significant emphasis has been placed on the advancement of learning shared knowledge from a source domain to a target domain. Recently, the large vision-language pre-trained model, i.e., CLIP has shown strong ability on zero-shot recognition, and parameter efficient tuning can further improve its performance on specific tasks. This work demonstrates that a simple domain prior boosts CLIP's zero-shot recognition in a specific domain. Besides, CLIP's adaptation relies less on source domain data due to its diverse pre-training dataset. Furthermore, we create a benchmark for zero-shot adaptation and pseudo-labeling based self-training with CLIP. Last but not least, we propose to improve the task generalization ability of CLIP from multiple unlabeled domains, which is a more practical and unique scenario. We believe our findings motivate a rethinking of domain adaptation benchmarks and the associated role of related algorithms in the era of CLIP.

Related papers

• In the Era of Prompt Learning with Vision-Language Models [1.060608983034705]
  We introduce textscStyLIP, a novel domain-agnostic prompt learning strategy for Domain Generalization (DG) StyLIP disentangles visual style and content in CLIPs vision encoder by using style projectors to learn domain-specific prompt tokens. We also propose AD-CLIP for unsupervised domain adaptation (DA), leveraging CLIPs frozen vision backbone.
  arXiv  Detail & Related papers  (2024-11-07T17:31:21Z)
• Learning to Discover Knowledge: A Weakly-Supervised Partial Domain Adaptation Approach [20.899013563493202]
  Domain adaptation has shown appealing performance by leveraging knowledge from a source domain with rich annotations. For a specific target task, it is cumbersome to collect related and high-quality source domains. In this paper, we propose a simple yet effective domain adaptation approach, termed as self-paced transfer classifier learning (SP-TCL)
  arXiv  Detail & Related papers  (2024-06-20T12:54:07Z)
• More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs [40.54076184225558]
  The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, known as Catastrophic Forgetting (CF) This paper presents a challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI) The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks.
  arXiv  Detail & Related papers  (2024-05-28T05:00:12Z)
• Unified Language-driven Zero-shot Domain Adaptation [55.64088594551629]
  Unified Language-driven Zero-shot Domain Adaptation (ULDA) is a novel task setting. It enables a single model to adapt to diverse target domains without explicit domain-ID knowledge.
  arXiv  Detail & Related papers  (2024-04-10T16:44:11Z)
• Open-Set Domain Adaptation with Visual-Language Foundation Models [51.49854335102149]
  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge from a source domain to a target domain with unlabeled data. Open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase.
  arXiv  Detail & Related papers  (2023-07-30T11:38:46Z)
• MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning [12.737883740101438]
  We present a novel DIL approach based on a mixture of prompt-tuned CLIP models (MoP-CLIP) At the training stage we model the features distribution of every class in each domain, learning individual text and visual prompts to adapt to a given domain. At inference, the learned distributions allow us to identify whether a given test sample belongs to a known domain, selecting the correct prompt for the classification task, or from an unseen domain.
  arXiv  Detail & Related papers  (2023-07-11T18:17:50Z)
• Domain Adaptation from Scratch [24.612696638386623]
  We present a new learning setup, domain adaptation from scratch'', which we believe to be crucial for extending the reach of NLP to sensitive domains. In this setup, we aim to efficiently annotate data from a set of source domains such that the trained model performs well on a sensitive target domain. Our study compares several approaches for this challenging setup, ranging from data selection and domain adaptation algorithms to active learning paradigms.
  arXiv  Detail & Related papers  (2022-09-02T05:55:09Z)
• Deep face recognition with clustering based domain adaptation [57.29464116557734]
  We propose a new clustering-based domain adaptation method designed for face recognition task in which the source and target domain do not share any classes. Our method effectively learns the discriminative target feature by aligning the feature domain globally, and, at the meantime, distinguishing the target clusters locally.
  arXiv  Detail & Related papers  (2022-05-27T12:29:11Z)
• Cross-domain Contrastive Learning for Unsupervised Domain Adaptation [108.63914324182984]
  Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a fully-labeled source domain to a different unlabeled target domain. We build upon contrastive self-supervised learning to align features so as to reduce the domain discrepancy between training and testing sets.
  arXiv  Detail & Related papers  (2021-06-10T06:32:30Z)
• Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation [74.71931918541748]
  We propose an instance affinity based criterion for source to target transfer during adaptation, called ILA-DA. We first propose a reliable and efficient method to extract similar and dissimilar samples across source and target, and utilize a multi-sample contrastive loss to drive the domain alignment process. We verify the effectiveness of ILA-DA by observing consistent improvements in accuracy over popular domain adaptation approaches on a variety of benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-03T01:33:14Z)
• Meta-Learning for Domain Generalization in Semantic Parsing [124.32975734073949]
  We use a meta-learning framework which targets zero-shot domain for semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing virtual train and test sets from disjoint domains.
  arXiv  Detail & Related papers  (2020-10-22T19:00:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.