Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models

• URL: http://arxiv.org/abs/2010.01739v1
• Date: Mon, 5 Oct 2020 01:49:47 GMT
• Title: Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models
• Authors: Thuy-Trang Vu, Dinh Phung and Gholamreza Haffari
• Abstract summary: We show that careful masking strategies can bridge the knowledge gap of masked language models. We propose an effective training strategy by adversarially masking out those tokens which are harder to adversarial by the underlying.
• Score: 54.569004548170824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has shown the importance of adaptation of broad-coverage contextualised embedding models on the domain of the target task of interest. Current self-supervised adaptation methods are simplistic, as the training signal comes from a small percentage of \emph{randomly} masked-out tokens. In this paper, we show that careful masking strategies can bridge the knowledge gap of masked language models (MLMs) about the domains more effectively by allocating self-supervision where it is needed. Furthermore, we propose an effective training strategy by adversarially masking out those tokens which are harder to reconstruct by the underlying MLM. The adversarial objective leads to a challenging combinatorial optimisation problem over \emph{subsets} of tokens, which we tackle efficiently through relaxation to a variational lowerbound and dynamic programming. On six unsupervised domain adaptation tasks involving named entity recognition, our method strongly outperforms the random masking strategy and achieves up to +1.64 F1 score improvements.

Related papers

• DAWN: Domain-Adaptive Weakly Supervised Nuclei Segmentation via Cross-Task Interactions [17.68742587885609]
  Current weakly supervised nuclei segmentation approaches follow a two-stage pseudo-label generation and network training process. This paper introduces a novel domain-adaptive weakly supervised nuclei segmentation framework using cross-task interaction strategies. To validate the effectiveness of our proposed method, we conduct extensive comparative and ablation experiments on six datasets.
  arXiv  Detail & Related papers  (2024-04-23T12:01:21Z)
• Emerging Property of Masked Token for Effective Pre-training [15.846621577804791]
  Masked Image Modeling (MIM) has been instrumental in driving recent breakthroughs in computer vision. MIM's overall efficiency is occasionally hampered by the lengthy duration of the pre-training phase. We propose a novel approach termed masked token optimization (MTO), specifically designed to improve model efficiency through weight recalibration and the enhancement of the key property of masked tokens.
  arXiv  Detail & Related papers  (2024-04-12T08:46:53Z)
• Mutual-modality Adversarial Attack with Semantic Perturbation [81.66172089175346]
  We propose a novel approach that generates adversarial attacks in a mutual-modality optimization scheme. Our approach outperforms state-of-the-art attack methods and can be readily deployed as a plug-and-play solution.
  arXiv  Detail & Related papers  (2023-12-20T05:06:01Z)
• Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation [49.827306773992376]
  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions. Our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-12-19T15:34:52Z)
• Boosting Adversarial Transferability with Learnable Patch-wise Masks [16.46210182214551]
  Adversarial examples have attracted widespread attention in security-critical applications because of their transferability across different models. In this paper, we argue that the model-specific discriminative regions are a key factor causing overfitting to the source model, and thus reducing the transferability to the target model. To accurately localize these regions, we present a learnable approach to automatically optimize the mask.
  arXiv  Detail & Related papers  (2023-06-28T05:32:22Z)
• Towards Fewer Annotations: Active Learning via Region Impurity and Prediction Uncertainty for Domain Adaptive Semantic Segmentation [19.55572909866489]
  We propose a region-based active learning approach for semantic segmentation under a domain shift. Our algorithm, Active Learning via Region Impurity and Prediction Uncertainty (AL-RIPU), introduces a novel acquisition strategy characterizing the spatial adjacency of image regions. Our method only requires very few annotations to almost reach the supervised performance and substantially outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-25T06:40:58Z)
• Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning [62.7588467386166]
  We leverage contrastive learning to bridge the domain gap by aligning the features of structurally similar label patches across domains. Our approach consistently outperforms state-of-the-art unsupervised and semi-supervised methods on two challenging domain adaptive segmentation tasks.
  arXiv  Detail & Related papers  (2021-04-22T13:39:12Z)
• Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation [63.195935452646815]
  We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training. We present a novel reinforcement learning-based framework which learns the masking policy. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets.
  arXiv  Detail & Related papers  (2020-10-06T13:27:01Z)
• Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension [96.62963688510035]
  Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation. We present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation.
  arXiv  Detail & Related papers  (2020-04-13T17:20:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.