AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

• URL: http://arxiv.org/abs/2311.00408v1
• Date: Wed, 1 Nov 2023 10:00:15 GMT
• Title: AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification
• Abstract summary: Few-shot sentence classification based on pre-trained Sentences (SEs) is efficient, robust, and effective. We propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs.
• Score: 53.84663977283108
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent's effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

Related papers

• Domain-Division based Progressive Learning for Source-Free Domain Adaptation [73.44692421206774]
  Source-free domain adaptation requires only a source pre-trained model and an unlabeled target domain.<n>We propose a domain-division based progressive learning method named DPL.<n>Our approach consists of two alternating stages, each beginning with the division of the target domain into easy-to-adapt and hard-to-adapt.
  arXiv  Detail & Related papers  (2026-07-31T09:21:56Z)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data [55.87500250831868]
  We study a simple strategy, specialized pretraining (SPT), where a small domain dataset is repeated starting from pretraining as a fraction of the total tokens.<n>In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x.<n>Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility.
  arXiv  Detail & Related papers  (2026-03-17T06:55:53Z)
• Is Large-Scale Pretraining the Secret to Good Domain Generalization? [69.80606575323691]
  Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP.
  arXiv  Detail & Related papers  (2024-12-03T21:43:11Z)
• The Last Mile to Supervised Performance: Semi-Supervised Domain Adaptation for Semantic Segmentation [51.77968964691317]
  We study the promising setting of Semi-Supervised Domain Adaptation (SSDA) We propose a simple SSDA framework that combines consistency regularization, pixel contrastive learning, and self-training to effectively utilize a few target-domain labels. Our method outperforms prior art in the popular GTA-to-Cityscapes benchmark and shows that as little as 50 target labels can suffice to achieve near-supervised performance.
  arXiv  Detail & Related papers  (2024-11-27T20:07:42Z)
• DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs [5.402030962296633]
  Early Exit (EE) strategies handle the issue by allowing the samples to exit from classifiers attached to intermediary layers. We propose Unsupervised Domain Adaptation in EE framework (DADEE) that employs multi-level adaptation using knowledge distillation. Experiments on tasks such as sentiment analysis, entailment classification, and natural language inference demonstrate that DADEE consistently outperforms early exit methods.
  arXiv  Detail & Related papers  (2024-10-06T09:44:58Z)
• How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation? [23.454153602068786]
  We evaluate the utility of Continued Pre-Training (CPT) for generative UDA. Our findings suggest that a implicitly learns the downstream task while predicting masked words informative to that task.
  arXiv  Detail & Related papers  (2024-01-31T00:15:34Z)
• Better Practices for Domain Adaptation [62.70267990659201]
  Domain adaptation (DA) aims to provide frameworks for adapting models to deployment data without using labels. Unclear validation protocol for DA has led to bad practices in the literature. We show challenges across all three branches of domain adaptation methodology.
  arXiv  Detail & Related papers  (2023-09-07T17:44:18Z)
• FDAPT: Federated Domain-adaptive Pre-training for Language Models [15.755622890097941]
  This paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT) We conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-Adaptive Pre-Training (FDAPT) We propose a novel algorithm, Frozen Federated Domain-Adaptive Pre-Training (FFDAPT)
  arXiv  Detail & Related papers  (2023-07-12T17:04:28Z)
• Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification [0.7874708385247353]
  We study the impact of training only the embedding layer on the model's performance during TAPT and task-specific finetuning. We show that training only the BERT embedding layer during TAPT is sufficient to adapt to the vocabulary of the target domain and achieve comparable performance.
  arXiv  Detail & Related papers  (2022-09-26T18:29:12Z)
• Domain Adaptation with Adversarial Training on Penultimate Activations [82.9977759320565]
  Enhancing model prediction confidence on unlabeled target data is an important objective in Unsupervised Domain Adaptation (UDA) We show that this strategy is more efficient and better correlated with the objective of boosting prediction confidence than adversarial training on input images or intermediate features.
  arXiv  Detail & Related papers  (2022-08-26T19:50:46Z)
• Re-energizing Domain Discriminator with Sample Relabeling for Adversarial Domain Adaptation [88.86865069583149]
  Unsupervised domain adaptation (UDA) methods exploit domain adversarial training to align the features to reduce domain gap. In this work, we propose an efficient optimization strategy named Re-enforceable Adversarial Domain Adaptation (RADA) RADA aims to re-energize the domain discriminator during the training by using dynamic domain labels.
  arXiv  Detail & Related papers  (2021-03-22T08:32:55Z)
• Effective Label Propagation for Discriminative Semi-Supervised Domain Adaptation [76.41664929948607]
  Semi-supervised domain adaptation (SSDA) methods have demonstrated great potential in large-scale image classification tasks. We present a novel and effective method to tackle this problem by using effective inter-domain and intra-domain semantic information propagation. Our source code and pre-trained models will be released soon.
  arXiv  Detail & Related papers  (2020-12-04T14:28:19Z)
• Rethinking Distributional Matching Based Domain Adaptation [111.15106414932413]
  Domain adaptation (DA) is a technique that transfers predictive models trained on a labeled source domain to an unlabeled target domain. Most popular DA algorithms are based on distributional matching (DM) In this paper, we first systematically analyze the limitations of DM based methods, and then build new benchmarks with more realistic domain shifts.
  arXiv  Detail & Related papers  (2020-06-23T21:55:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.