Domain Adversarial Fine-Tuning as an Effective Regularizer
- URL: http://arxiv.org/abs/2009.13366v2
- Date: Mon, 5 Oct 2020 23:56:19 GMT
- Title: Domain Adversarial Fine-Tuning as an Effective Regularizer
- Authors: Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, Ion
Androutsopoulos
- Abstract summary: In Natural Language Processing (NLP), pretrained language models (LMs) that are transferred to downstream tasks have been recently shown to achieve state-of-the-art results.
Standard fine-tuning can degrade the general-domain representations captured during pretraining.
We introduce a new regularization technique, AFTER; domain Adversarial Fine-Tuning as an Effective Regularizer.
- Score: 80.14528207465412
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In Natural Language Processing (NLP), pretrained language models (LMs) that
are transferred to downstream tasks have been recently shown to achieve
state-of-the-art results. However, standard fine-tuning can degrade the
general-domain representations captured during pretraining. To address this
issue, we introduce a new regularization technique, AFTER; domain Adversarial
Fine-Tuning as an Effective Regularizer. Specifically, we complement the
task-specific loss used during fine-tuning with an adversarial objective. This
additional loss term is related to an adversarial classifier, that aims to
discriminate between in-domain and out-of-domain text representations.
In-domain refers to the labeled dataset of the task at hand while out-of-domain
refers to unlabeled data from a different domain. Intuitively, the adversarial
classifier acts as a regularizer which prevents the model from overfitting to
the task-specific domain. Empirical results on various natural language
understanding tasks show that AFTER leads to improved performance compared to
standard fine-tuning.
Related papers
- Regex-augmented Domain Transfer Topic Classification based on a
Pre-trained Language Model: An application in Financial Domain [42.5087655999509]
We discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning.
Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks.
arXiv Detail & Related papers (2023-05-23T03:26:32Z) - Cyclically Disentangled Feature Translation for Face Anti-spoofing [61.70377630461084]
We propose a novel domain adaptation method called cyclically disentangled feature translation network (CDFTN)
CDFTN generates pseudo-labeled samples that possess: 1) source domain-invariant liveness features and 2) target domain-specific content features, which are disentangled through domain adversarial training.
A robust classifier is trained based on the synthetic pseudo-labeled images under the supervision of source domain labels.
arXiv Detail & Related papers (2022-12-07T14:12:34Z) - Domain Classification-based Source-specific Term Penalization for Domain
Adaptation in Hate-speech Detection [30.462596705180534]
State-of-the-art approaches for hate-speech detection exhibit poor performance in out-of-domain settings.
We propose a domain adaptation approach that automatically extracts and penalizes source-specific terms.
arXiv Detail & Related papers (2022-09-18T23:52:22Z) - Task Transfer and Domain Adaptation for Zero-Shot Question Answering [18.188082154309175]
We use supervised pretraining on source-domain data to reduce sample complexity on domain-specific downstream tasks.
We evaluate zero-shot performance on domain-specific reading comprehension tasks by combining task transfer with domain adaptation.
arXiv Detail & Related papers (2022-06-14T09:10:48Z) - Domain-Agnostic Prior for Transfer Semantic Segmentation [197.9378107222422]
Unsupervised domain adaptation (UDA) is an important topic in the computer vision community.
We present a mechanism that regularizes cross-domain representation learning with a domain-agnostic prior (DAP)
Our research reveals that UDA benefits much from better proxies, possibly from other data modalities.
arXiv Detail & Related papers (2022-04-06T09:13:25Z) - Task-specific Inconsistency Alignment for Domain Adaptive Object
Detection [38.027790951157705]
Detectors trained with massive labeled data often exhibit dramatic performance degradation in certain scenarios with data distribution gap.
We propose Task-specific Inconsistency Alignment (TIA), by developing a new alignment mechanism in separate task spaces.
TIA demonstrates superior results on various scenarios to the previous state-of-the-art methods.
arXiv Detail & Related papers (2022-03-29T08:36:33Z) - DILBERT: Customized Pre-Training for Domain Adaptation withCategory
Shift, with an Application to Aspect Extraction [25.075552473110676]
A generic approach towards the pre-training procedure can naturally be sub-optimal in some cases.
This paper presents a new fine-tuning scheme for BERT, which aims to address the above challenges.
We name this scheme DILBERT: Domain Invariant Learning with BERT, and customize it for aspect extraction in the unsupervised domain adaptation setting.
arXiv Detail & Related papers (2021-09-01T18:49:44Z) - Contrastive Learning and Self-Training for Unsupervised Domain
Adaptation in Semantic Segmentation [71.77083272602525]
UDA attempts to provide efficient knowledge transfer from a labeled source domain to an unlabeled target domain.
We propose a contrastive learning approach that adapts category-wise centroids across domains.
We extend our method with self-training, where we use a memory-efficient temporal ensemble to generate consistent and reliable pseudo-labels.
arXiv Detail & Related papers (2021-05-05T11:55:53Z) - Re-energizing Domain Discriminator with Sample Relabeling for
Adversarial Domain Adaptation [88.86865069583149]
Unsupervised domain adaptation (UDA) methods exploit domain adversarial training to align the features to reduce domain gap.
In this work, we propose an efficient optimization strategy named Re-enforceable Adversarial Domain Adaptation (RADA)
RADA aims to re-energize the domain discriminator during the training by using dynamic domain labels.
arXiv Detail & Related papers (2021-03-22T08:32:55Z) - Alleviating Semantic-level Shift: A Semi-supervised Domain Adaptation
Method for Semantic Segmentation [97.8552697905657]
A key challenge of this task is how to alleviate the data distribution discrepancy between the source and target domains.
We propose Alleviating Semantic-level Shift (ASS), which can successfully promote the distribution consistency from both global and local views.
We apply our ASS to two domain adaptation tasks, from GTA5 to Cityscapes and from Synthia to Cityscapes.
arXiv Detail & Related papers (2020-04-02T03:25:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.