UDALM: Unsupervised Domain Adaptation through Language Modeling

• URL: http://arxiv.org/abs/2104.07078v1
• Date: Wed, 14 Apr 2021 19:05:01 GMT
• Title: UDALM: Unsupervised Domain Adaptation through Language Modeling
• Authors: Constantinos Karouzos, Georgios Paraskevopoulos and Alexandros Potamianos
• Abstract summary: We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data can be effectively used as a stopping criterion. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding $91.74%$ accuracy, which is an $1.11%$ absolute improvement over the state-of-versathe-art.
• Score: 79.73916345178415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding $91.74\%$ accuracy, which is an $1.11\%$ absolute improvement over the state-of-the-art.

Related papers

• Stratified Domain Adaptation: A Progressive Self-Training Approach for Scene Text Recognition [1.2878987353423252]
  Unsupervised domain adaptation (UDA) has become increasingly prevalent in scene text recognition (STR) We introduce the Stratified Domain Adaptation (StrDA) approach, which examines the gradual escalation of the domain gap for the learning process. We propose a novel method for employing domain discriminators to estimate the out-of-distribution and domain discriminative levels of data samples.
  arXiv  Detail & Related papers  (2024-10-13T16:40:48Z)
• Unsupervised Domain Adaptation Via Data Pruning [0.0]
  We consider the problem from the perspective of unsupervised domain adaptation (UDA) We propose AdaPrune, a method for UDA whereby training examples are removed to attempt to align the training distribution to that of the target data. As a method for UDA, we show that AdaPrune outperforms related techniques, and is complementary to other UDA algorithms such as CORAL.
  arXiv  Detail & Related papers  (2024-09-18T15:48:59Z)
• Open-Set Domain Adaptation with Visual-Language Foundation Models [51.49854335102149]
  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge from a source domain to a target domain with unlabeled data. Open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase.
  arXiv  Detail & Related papers  (2023-07-30T11:38:46Z)
• Divide and Adapt: Active Domain Adaptation via Customized Learning [56.79144758380419]
  We present Divide-and-Adapt (DiaNA), a new ADA framework that partitions the target instances into four categories with stratified transferable properties. With a novel data subdivision protocol based on uncertainty and domainness, DiaNA can accurately recognize the most gainful samples. Thanks to the "divideand-adapt" spirit, DiaNA can handle data with large variations of domain gap.
  arXiv  Detail & Related papers  (2023-07-21T14:37:17Z)
• Can We Evaluate Domain Adaptation Models Without Target-Domain Labels? [36.05871459064825]
  Unsupervised domain adaptation (UDA) involves adapting a model trained on a label-rich source domain to an unlabeled target domain. In real-world scenarios, the absence of target-domain labels makes it challenging to evaluate the performance of UDA models. We propose a novel metric called the textitTransfer Score to address these issues.
  arXiv  Detail & Related papers  (2023-05-30T03:36:40Z)
• IDA: Informed Domain Adaptive Semantic Segmentation [51.12107564372869]
  We propose an Domain Informed Adaptation (IDA) model, a self-training framework that mixes the data based on class-level segmentation performance. In our IDA model, the class-level performance is tracked by an expected confidence score (ECS) and we then use a dynamic schedule to determine the mixing ratio for data in different domains. Our proposed method is able to outperform the state-of-the-art UDA-SS method by a margin of 1.1 mIoU in the adaptation of GTA-V to Cityscapes and of 0.9 mIoU in the adaptation of SYNTHIA to City
  arXiv  Detail & Related papers  (2023-03-05T18:16:34Z)
• Adversarial Robustness for Unsupervised Domain Adaptation [48.51898925429575]
  In this work, we leverage intermediate representations learned by multiple robust ImageNet models to improve the robustness of UDA models. Our method works by aligning the features of the UDA model with the robust features learned by ImageNet pre-trained models along with domain adaptation training.
  arXiv  Detail & Related papers  (2021-09-02T13:45:01Z)
• On Universal Black-Box Domain Adaptation [53.7611757926922]
  We study an arguably least restrictive setting of domain adaptation in a sense of practical deployment. Only the interface of source model is available to the target domain, and where the label-space relations between the two domains are allowed to be different and unknown. We propose to unify them into a self-training framework, regularized by consistency of predictions in local neighborhoods of target samples.
  arXiv  Detail & Related papers  (2021-04-10T02:21:09Z)
• Robustified Domain Adaptation [13.14535125302501]
  Unsupervised domain adaptation (UDA) is widely used to transfer knowledge from a labeled source domain to an unlabeled target domain. The inevitable domain distribution deviation in UDA is a critical barrier to model robustness on the target domain. We propose a novel Class-consistent Unsupervised Domain Adaptation (CURDA) framework for training robust UDA models.
  arXiv  Detail & Related papers  (2020-11-18T22:21:54Z)
• Knowledge Distillation for BERT Unsupervised Domain Adaptation [2.969705152497174]
  A pre-trained language model, BERT, has brought significant performance improvements across a range of natural language processing tasks. We propose a simple but effective unsupervised domain adaptation method, adversarial adaptation with distillation (AAD) We evaluate our approach in the task of cross-domain sentiment classification on 30 domain pairs.
  arXiv  Detail & Related papers  (2020-10-22T06:51:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.