Learning to Adapt to Low-Resource Paraphrase Generation

• URL: http://arxiv.org/abs/2412.17111v1
• Date: Sun, 22 Dec 2024 17:55:52 GMT
• Title: Learning to Adapt to Low-Resource Paraphrase Generation
• Authors: Zhigen Li, Yanmeng Wang, Rizhao Fan, Ye Wang, Jianfeng Li, Shaojun Wang,
• Abstract summary: Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora. transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse. At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data. We propose LAPA, an effective adapter for PLMs optimized by meta-learning.
• Score: 15.318954551770204
• License:
• Abstract: Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora. However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse. At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data. To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning. LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data. This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task. Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets. With only 2\% of trainable parameters and 1\% labeled data of the target task, our approach can achieve a competitive performance with previous work.

Related papers

• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Few-shot learning for automated content analysis: Efficient coding of arguments and claims in the debate on arms deliveries to Ukraine [0.9576975587953563]
  Pre-trained language models (PLM) based on transformer neural networks offer great opportunities to improve automatic content analysis in communication science. Three characteristics so far impeded the widespread adoption of the methods in the applying disciplines: the dominance of English language models in NLP research, the necessary computing resources, and the effort required to produce training data to fine-tune PLMs. We test our approach on a realistic use case from communication science to automatically detect claims and arguments together with their stance in the German news debate on arms deliveries to Ukraine.
  arXiv  Detail & Related papers  (2023-12-28T11:39:08Z)
• LLMaAA: Making Large Language Models as Active Annotators [32.57011151031332]
  We propose LLMaAA, which takes large language models as annotators and puts them into an active learning loop to determine what to annotate efficiently. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples.
  arXiv  Detail & Related papers  (2023-10-30T14:54:15Z)
• Learning to Paraphrase Sentences to Different Complexity Levels [3.0273878903284275]
  Sentence simplification is an active research topic in NLP, but its adjacent tasks of sentence complexification and same-level paraphrasing are not. To train models on all three tasks, we present two new unsupervised datasets.
  arXiv  Detail & Related papers  (2023-08-04T09:43:37Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study [76.52997424694767]
  We present an in-depth empirical study of keyphrase extraction and keyphrase generation using pre-trained language models. We show that PLMs have competitive high-resource performance and state-of-the-art low-resource performance. Further results show that in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models.
  arXiv  Detail & Related papers  (2022-12-20T13:20:21Z)
• Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding [62.17020485045456]
  It is commonly assumed in semi-supervised learning (SSL) that the unlabeled data are drawn from the same distribution as that of the labeled ones. We propose to selectively utilize unlabeled data through sample weighting, so that only conducive unlabeled data would be prioritized.
  arXiv  Detail & Related papers  (2022-05-02T16:09:17Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)
• Multi-Stage Pre-training for Low-Resource Domain Adaptation [24.689862495171408]
  Current approaches directly adapt a pre-trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain.
  arXiv  Detail & Related papers  (2020-10-12T17:57:00Z)
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks [81.99843216550306]
  We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks. A second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains. Adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining.
  arXiv  Detail & Related papers  (2020-04-23T04:21:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.