Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units

• URL: http://arxiv.org/abs/2106.04935v1
• Date: Wed, 9 Jun 2021 09:29:11 GMT
• Title: Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
• Authors: Sara Meftah, Nasredine Semmar, Youssef Tamaazousti, Hassane Essafi, Fatiha Sadat
• Abstract summary: Neural Transfer Learning (TL) is becoming ubiquitous in Natural Language Processing (NLP) In this paper, we show through interpretation methods that such scheme, despite its efficiency, is suffering from a main limitation. We propose to augment the pre-trained model with normalised, weighted and randomly initialised units that foster a better adaptation while maintaining the valuable source knowledge.
• Score: 14.183224769428843
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Transfer Learning (TL) is becoming ubiquitous in Natural Language Processing (NLP), thanks to its high performance on many tasks, especially in low-resourced scenarios. Notably, TL is widely used for neural domain adaptation to transfer valuable knowledge from high-resource to low-resource domains. In the standard fine-tuning scheme of TL, a model is initially pre-trained on a source domain and subsequently fine-tuned on a target domain and, therefore, source and target domains are trained using the same architecture. In this paper, we show through interpretation methods that such scheme, despite its efficiency, is suffering from a main limitation. Indeed, although capable of adapting to new domains, pre-trained neurons struggle with learning certain patterns that are specific to the target domain. Moreover, we shed light on the hidden negative transfer occurring despite the high relatedness between source and target domains, which may mitigate the final gain brought by transfer learning. To address these problems, we propose to augment the pre-trained model with normalised, weighted and randomly initialised units that foster a better adaptation while maintaining the valuable source knowledge. We show that our approach exhibits significant improvements to the standard fine-tuning scheme for neural domain adaptation from the news domain to the social media domain on four NLP tasks: part-of-speech tagging, chunking, named entity recognition and morphosyntactic tagging.

Related papers

• Learning to Discover Knowledge: A Weakly-Supervised Partial Domain Adaptation Approach [20.899013563493202]
  Domain adaptation has shown appealing performance by leveraging knowledge from a source domain with rich annotations. For a specific target task, it is cumbersome to collect related and high-quality source domains. In this paper, we propose a simple yet effective domain adaptation approach, termed as self-paced transfer classifier learning (SP-TCL)
  arXiv  Detail & Related papers  (2024-06-20T12:54:07Z)
• Progressive Conservative Adaptation for Evolving Target Domains [76.9274842289221]
  Conventional domain adaptation typically transfers knowledge from a source domain to a stationary target domain. Restoring and adapting to such target data results in escalating computational and resource consumption over time. We propose a simple yet effective approach, termed progressive conservative adaptation (PCAda)
  arXiv  Detail & Related papers  (2024-02-07T04:11:25Z)
• Domain Adaptation from Scratch [24.612696638386623]
  We present a new learning setup, domain adaptation from scratch'', which we believe to be crucial for extending the reach of NLP to sensitive domains. In this setup, we aim to efficiently annotate data from a set of source domains such that the trained model performs well on a sensitive target domain. Our study compares several approaches for this challenging setup, ranging from data selection and domain adaptation algorithms to active learning paradigms.
  arXiv  Detail & Related papers  (2022-09-02T05:55:09Z)
• RAIN: RegulArization on Input and Network for Black-Box Domain Adaptation [80.03883315743715]
  Source-free domain adaptation transits the source-trained model towards target domain without exposing the source data. This paradigm is still at risk of data leakage due to adversarial attacks on the source model. We propose a novel approach named RAIN (RegulArization on Input and Network) for Black-Box domain adaptation from both input-level and network-level regularization.
  arXiv  Detail & Related papers  (2022-08-22T18:18:47Z)
• DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction [25.075552473110676]
  A generic approach towards the pre-training procedure can naturally be sub-optimal in some cases. This paper presents a new fine-tuning scheme for BERT, which aims to address the above challenges. We name this scheme DILBERT: Domain Invariant Learning with BERT, and customize it for aspect extraction in the unsupervised domain adaptation setting.
  arXiv  Detail & Related papers  (2021-09-01T18:49:44Z)
• Gradient Regularized Contrastive Learning for Continual Domain Adaptation [86.02012896014095]
  We study the problem of continual domain adaptation, where the model is presented with a labelled source domain and a sequence of unlabelled target domains. We propose Gradient Regularized Contrastive Learning (GRCL) to solve the obstacles. Experiments on Digits, DomainNet and Office-Caltech benchmarks demonstrate the strong performance of our approach.
  arXiv  Detail & Related papers  (2021-03-23T04:10:42Z)
• Domain Adaptation for Semantic Parsing [68.81787666086554]
  We propose a novel semantic for domain adaptation, where we have much fewer annotated data in the target domain compared to the source domain. Our semantic benefits from a two-stage coarse-to-fine framework, thus can provide different and accurate treatments for the two stages. Experiments on a benchmark dataset show that our method consistently outperforms several popular domain adaptation strategies.
  arXiv  Detail & Related papers  (2020-06-23T14:47:41Z)
• Neural Unsupervised Domain Adaptation in NLP---A Survey [23.104354433276246]
  We review neural unsupervised domain adaptation techniques which do not require labeled target domain data. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention.
  arXiv  Detail & Related papers  (2020-05-31T22:34:14Z)
• Vocabulary Adaptation for Distant Domain Adaptation in Neural Machine Translation [14.390932594872233]
  Domain adaptation between distant domains cannot be performed effectively due to mismatches in vocabulary. We propose vocabulary adaptation, a simple method for effective fine-tuning. Our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En-Ja and De-En translation.
  arXiv  Detail & Related papers  (2020-04-30T14:27:59Z)
• Deep Residual Correction Network for Partial Domain Adaptation [79.27753273651747]
  Deep domain adaptation methods have achieved appealing performance by learning transferable representations from a well-labeled source domain to a different but related unlabeled target domain. This paper proposes an efficiently-implemented Deep Residual Correction Network (DRCN) Comprehensive experiments on partial, traditional and fine-grained cross-domain visual recognition demonstrate that DRCN is superior to the competitive deep domain adaptation approaches.
  arXiv  Detail & Related papers  (2020-04-10T06:07:16Z)
• Supervised Domain Adaptation using Graph Embedding [86.3361797111839]
  Domain adaptation methods assume that distributions between the two domains are shifted and attempt to realign them. We propose a generic framework based on graph embedding. We show that the proposed approach leads to a powerful Domain Adaptation framework.
  arXiv  Detail & Related papers  (2020-03-09T12:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.