Gradual Fine-Tuning for Low-Resource Domain Adaptation

• URL: http://arxiv.org/abs/2103.02205v1
• Date: Wed, 3 Mar 2021 06:24:54 GMT
• Title: Gradual Fine-Tuning for Low-Resource Domain Adaptation
• Authors: Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme and Kenton Murray
• Abstract summary: Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. We demonstrate that gradually fine-tuning in a multi-stage process can yield substantial further gains and can be applied without modifying the model or learning objective.
• Score: 33.80484557176643
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. Such domain adaptation is typically done using one stage of fine-tuning. We demonstrate that gradually fine-tuning in a multi-stage process can yield substantial further gains and can be applied without modifying the model or learning objective.

Related papers

• Domain Generalization Guided by Large-Scale Pre-Trained Priors [24.74398777539288]
  Domain generalization (DG) aims to train a model from limited source domains, allowing it to generalize to unknown target domains. We introduce Fine-Tune with Large-scale pre-trained Priors (FT-LP) FT-LP incorporates the pre-trained model as a prior into the DG fine-tuning process, ensuring that the model refers to its pre-trained model at each optimization step.
  arXiv  Detail & Related papers  (2024-06-09T03:32:32Z)
• Mitigate Domain Shift by Primary-Auxiliary Objectives Association for Generalizing Person ReID [39.98444065846305]
  ReID models struggle in learning domain-invariant representation solely through training on an instance classification objective. We introduce a method that guides model learning of the primary ReID instance classification objective by a concurrent auxiliary learning objective on weakly labeled pedestrian saliency detection. Our model can be extended with the recent test-time diagram to form the PAOA+, which performs on-the-fly optimization against the auxiliary objective.
  arXiv  Detail & Related papers  (2023-10-24T15:15:57Z)
• FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning [21.693779973263172]
  In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align) Our method aims to bolster the model's generalizability by preserving the consistency of spurious features. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements.
  arXiv  Detail & Related papers  (2023-10-23T17:12:01Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models [137.74524357614285]
  We introduce a novel Gradient-RegulAted Meta-prompt learning framework. It helps pre-training models adapt to downstream tasks in a parameter -- and data -- efficient way. GRAM can be easily incorporated into various prompt tuning methods in a model-agnostic way.
  arXiv  Detail & Related papers  (2023-03-12T05:03:37Z)
• Normalization Perturbation: A Simple Domain Generalization Method for Real-World Domain Shifts [133.99270341855728]
  Real-world domain styles can vary substantially due to environment changes and sensor noises. Deep models only know the training domain style. We propose Normalization Perturbation to overcome this domain style overfitting problem.
  arXiv  Detail & Related papers  (2022-11-08T17:36:49Z)
• On Fine-Tuned Deep Features for Unsupervised Domain Adaptation [23.18781318003242]
  We study the potential of combining fine-tuned features and feature transformation based UDA methods for improved domain adaptation performance. Specifically, we integrate the prevalent progressive pseudo-labelling techniques into the fine-tuning framework to extract fine-tuned features. Experiments with multiple deep models including ResNet-50/101 and DeiT-small/base are conducted to demonstrate the combination of fine-tuned features.
  arXiv  Detail & Related papers  (2022-10-25T15:07:04Z)
• Variational Model Perturbation for Source-Free Domain Adaptation [64.98560348412518]
  We introduce perturbations into the model parameters by variational Bayesian inference in a probabilistic framework. We demonstrate the theoretical connection to learning Bayesian neural networks, which proves the generalizability of the perturbed model to target domains.
  arXiv  Detail & Related papers  (2022-10-19T08:41:19Z)
• A Novel Mix-normalization Method for Generalizable Multi-source Person Re-identification [49.548815417844786]
  Person re-identification (Re-ID) has achieved great success in the supervised scenario. It is difficult to directly transfer the supervised model to arbitrary unseen domains due to the model overfitting to the seen source domains. We propose MixNorm, which consists of domain-aware mix-normalization (DMN) and domain-ware center regularization (DCR)
  arXiv  Detail & Related papers  (2022-01-24T18:09:38Z)
• PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models [20.62501560076402]
  PERL: A representation learning model that extends contextualized word embedding models such as BERT with pivot-based fine-tuning. PerL outperforms strong baselines across 22 sentiment classification domain adaptation setups. It yields effective reduced-size models and increases model stability.
  arXiv  Detail & Related papers  (2020-06-16T11:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.