Empowering Source-Free Domain Adaptation with MLLM-driven Curriculum Learning

• URL: http://arxiv.org/abs/2405.18376v1
• Date: Tue, 28 May 2024 17:18:17 GMT
• Title: Empowering Source-Free Domain Adaptation with MLLM-driven Curriculum Learning
• Authors: Dongjie Chen, Kartik Patwari, Zhengfeng Lai, Sen-ching Cheung, Chen-Nee Chuah,
• Abstract summary: Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to a target domain using only unlabeled target data. Reliability-based Curriculum Learning (RCL) integrates multiple MLLMs for knowledge exploitation via pseudo-labeling in SFDA.
• Score: 5.599218556731767
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to a target domain using only unlabeled target data. Current SFDA methods face challenges in effectively leveraging pre-trained knowledge and exploiting target domain data. Multimodal Large Language Models (MLLMs) offer remarkable capabilities in understanding visual and textual information, but their applicability to SFDA poses challenges such as instruction-following failures, intensive computational demands, and difficulties in performance measurement prior to adaptation. To alleviate these issues, we propose Reliability-based Curriculum Learning (RCL), a novel framework that integrates multiple MLLMs for knowledge exploitation via pseudo-labeling in SFDA. Our framework incorporates proposed Reliable Knowledge Transfer, Self-correcting and MLLM-guided Knowledge Expansion, and Multi-hot Masking Refinement to progressively exploit unlabeled data in the target domain. RCL achieves state-of-the-art (SOTA) performance on multiple SFDA benchmarks, e.g., $\textbf{+9.4%}$ on DomainNet, demonstrating its effectiveness in enhancing adaptability and robustness without requiring access to source data. Code: https://github.com/Dong-Jie-Chen/RCL.

Related papers

• Learn from the Learnt: Source-Free Active Domain Adaptation via Contrastive Sampling and Visual Persistence [60.37934652213881]
  Domain Adaptation (DA) facilitates knowledge transfer from a source domain to a related target domain. This paper investigates a practical DA paradigm, namely Source data-Free Active Domain Adaptation (SFADA), where source data becomes inaccessible during adaptation. We present learn from the learnt (LFTL), a novel paradigm for SFADA to leverage the learnt knowledge from the source pretrained model and actively iterated models without extra overhead.
  arXiv  Detail & Related papers  (2024-07-26T17:51:58Z)
• Memory-Efficient Pseudo-Labeling for Online Source-Free Universal Domain Adaptation using a Gaussian Mixture Model [3.1265626879839923]
  In practice, domain shifts are likely to occur between training and test data, necessitating domain adaptation (DA) to adjust the pre-trained source model to the target domain. UniDA has gained attention for addressing the possibility of an additional category (label) shift between the source and target domain. We propose a novel method that continuously captures the distribution of known classes in the feature space using a Gaussian mixture model (GMM) Our approach achieves state-of-the-art results in all experiments on the DomainNet and Office-Home datasets.
  arXiv  Detail & Related papers  (2024-07-19T11:13:31Z)
• CLAIM Your Data: Enhancing Imputation Accuracy with Contextual Large Language Models [0.18416014644193068]
  This paper introduces the Contextual Language model for Accurate Imputation Method (CLAIM) Unlike traditional imputation methods, CLAIM utilizes contextually relevant natural language descriptors to fill missing values. Our evaluations across diverse datasets and missingness patterns reveal CLAIM's superior performance over existing imputation techniques.
  arXiv  Detail & Related papers  (2024-05-28T00:08:29Z)
• Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Training [23.56208527227504]
  Source-free domain adaptation (SFDA) aims to adapt a source model trained on a fully-labeled source domain to a related but unlabeled target domain. In the conventional SFDA pipeline, a large data (e.g. ImageNet) pre-trained feature extractor is used to initialize the source model. We introduce an integrated framework to incorporate pre-trained networks into the target adaptation process.
  arXiv  Detail & Related papers  (2024-05-05T14:48:13Z)
• Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning [55.715623885418815]
  Cross-Domain Few-Shot Learning methods require access to source domain data to train a model in the pre-training phase. Due to increasing concerns about data privacy and the desire to reduce data transmission and training costs, it is necessary to develop a CDFSL solution without accessing source data. This paper proposes an Enhanced Information Maximization with Distance-Aware Contrastive Learning method to address these challenges.
  arXiv  Detail & Related papers  (2024-03-04T12:10:24Z)
• The Unreasonable Effectiveness of Large Language-Vision Models for Source-free Video Domain Adaptation [56.61543110071199]
  Source-Free Video Unsupervised Domain Adaptation (SFVUDA) task consists in adapting an action recognition model, trained on a labelled source dataset, to an unlabelled target dataset. Previous approaches have attempted to address SFVUDA by leveraging self-supervision derived from the target data itself. We take an approach by exploiting "web-supervision" from Large Language-Vision Models (LLVMs), driven by the rationale that LLVMs contain a rich world prior surprisingly robust to domain-shift.
  arXiv  Detail & Related papers  (2023-08-17T18:12:05Z)
• Open-Set Domain Adaptation with Visual-Language Foundation Models [51.49854335102149]
  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge from a source domain to a target domain with unlabeled data. Open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase.
  arXiv  Detail & Related papers  (2023-07-30T11:38:46Z)
• C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptation [18.58469201869616]
  Unsupervised domain adaptation (UDA) approaches focus on adapting models trained on a labeled source domain to an unlabeled target domain. We propose C-SFDA, a curriculum learning aided self-training framework for SFDA that adapts efficiently and reliably to changes across domains based on selective pseudo-labeling.
  arXiv  Detail & Related papers  (2023-03-30T03:42:54Z)
• KALA: Knowledge-Augmented Language Model Adaptation [65.92457495576141]
  We propose a novel domain adaption framework for pre-trained language models (PLMs) Knowledge-Augmented Language model Adaptation (KALA) modulates the intermediate hidden representations of PLMs with domain knowledge. Results show that, despite being computationally efficient, our KALA largely outperforms adaptive pre-training.
  arXiv  Detail & Related papers  (2022-04-22T08:11:59Z)
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation [102.67010690592011]
  Unsupervised adaptationUDA (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Prior UDA methods typically require to access the source data when learning to adapt the model. This work tackles a practical setting where only a trained source model is available and how we can effectively utilize such a model without source data to solve UDA problems.
  arXiv  Detail & Related papers  (2020-02-20T03:13:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.