Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization

• URL: http://arxiv.org/abs/2112.11670v1
• Date: Wed, 22 Dec 2021 05:34:56 GMT
• Title: Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization
• Authors: Md Tahmid Rahman Laskar, Enamul Hoque, Jimmy Xiangji Huang
• Abstract summary: The Query Focused Text Summarization (QFTS) task aims at building systems that generate the summary of the text document(s) based on a given query. A key challenge in addressing this task is the lack of large labeled data for training the summarization model. We address this challenge by exploring a series of domain adaptation techniques.
• Score: 18.791701342934605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Query Focused Text Summarization (QFTS) task aims at building systems that generate the summary of the text document(s) based on the given query. A key challenge in addressing this task is the lack of large labeled data for training the summarization model. In this paper, we address this challenge by exploring a series of domain adaptation techniques. Given the recent success of pre-trained transformer models in a wide range of natural language processing tasks, we utilize such models to generate abstractive summaries for the QFTS task for both single-document and multi-document scenarios. For domain adaptation, we apply a variety of techniques using pre-trained transformer-based summarization models including transfer learning, weakly supervised learning, and distant supervision. Extensive experiments on six datasets show that our proposed approach is very effective in generating abstractive summaries for the QFTS task while setting a new state-of-the-art result in several datasets across a set of automatic and human evaluation metrics.

Related papers

• Controllable Topic-Focused Abstractive Summarization [57.8015120583044]
  Controlled abstractive summarization focuses on producing condensed versions of a source article to cover specific aspects. This paper presents a new Transformer-based architecture capable of producing topic-focused summaries.
  arXiv  Detail & Related papers  (2023-11-12T03:51:38Z)
• Integrating Contrastive Learning into a Multitask Transformer Model for Effective Domain Adaptation [4.157415305926585]
  We propose a novel domain adaptation technique that embodies a multitask framework with SER as the primary task. We show that our proposed model achieves state-of-the-art performance in SER within cross-corpus scenarios.
  arXiv  Detail & Related papers  (2023-10-07T06:41:29Z)
• Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization [1.066048003460524]
  Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. Abstractive MDS aims to generate a coherent and fluent summary for multiple documents using natural language generation techniques. We propose Absformer, a new Transformer-based method for unsupervised abstractive summary generation.
  arXiv  Detail & Related papers  (2023-06-07T21:18:23Z)
• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning [101.66860222415512]
  Multi-Task Diffusion Model (textscMTDiff) is a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis. For generative planning, we find textscMTDiff outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D.
  arXiv  Detail & Related papers  (2023-05-29T05:20:38Z)
• Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering [49.85790367128085]
  We pre-training a generic multi-document model from a novel cross-document question answering pre-training objective. This novel multi-document QA formulation directs the model to better recover cross-text informational relations. Unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation and long text generation.
  arXiv  Detail & Related papers  (2023-05-24T17:48:40Z)
• Transformer Based Implementation for Automatic Book Summarization [0.0]
  Document Summarization is the procedure of generating a meaningful and concise summary of a given document. This work is an attempt to use Transformer based techniques for Abstract generation.
  arXiv  Detail & Related papers  (2023-01-17T18:18:51Z)
• Exploring Efficient Few-shot Adaptation for Vision Transformers [70.91692521825405]
  We propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViTs in the Few-shot Learning tasks. Key novelties come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) We conduct extensive experiments to show the efficacy of our model.
  arXiv  Detail & Related papers  (2023-01-06T08:42:05Z)
• DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization [14.414693156937782]
  We propose a novel domain-specific generative pre-training (DS-GPT) method for text generation. We apply it to the product titleand review summarization problems on E-commerce mobile display.
  arXiv  Detail & Related papers  (2021-12-15T19:02:49Z)
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization [57.798070356553936]
  HETFORMER is a Transformer-based pre-trained model with multi-granularity sparse attentions for extractive summarization. Experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1.
  arXiv  Detail & Related papers  (2021-10-12T22:42:31Z)
• WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization [16.048329028104643]
  In the Query Focused Multi-Document Summarization (QF-MDS) task, a set of documents and a query are given where the goal is to generate a summary from these documents. One major challenge for this task is the lack of availability of labeled training datasets. We propose a novel weakly supervised learning approach via utilizing distant supervision.
  arXiv  Detail & Related papers  (2020-11-03T02:02:55Z)
• Pretrained Transformers for Text Ranking: BERT and Beyond [53.83210899683987]
  This survey provides an overview of text ranking with neural network architectures known as transformers. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing.
  arXiv  Detail & Related papers  (2020-10-13T15:20:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.