Document-aware Positional Encoding and Linguistic-guided Encoding for Abstractive Multi-document Summarization

• URL: http://arxiv.org/abs/2209.05929v1
• Date: Tue, 13 Sep 2022 12:22:38 GMT
• Title: Document-aware Positional Encoding and Linguistic-guided Encoding for Abstractive Multi-document Summarization
• Authors: Congbo Ma, Wei Emma Zhang, Pitawelayalage Dasun Dileepa Pitawela, Yutong Qu, Haojie Zhuang, Hu Wang
• Abstract summary: One key challenge in multi-document summarization is to capture the relations among input documents that distinguish between single document summarization (SDS) and multi-document summarization (MDS) We propose document-aware positional encoding and linguistic-guided encoding that can be fused with Transformer architecture for MDS.
• Score: 12.799359904396624
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One key challenge in multi-document summarization is to capture the relations among input documents that distinguish between single document summarization (SDS) and multi-document summarization (MDS). Few existing MDS works address this issue. One effective way is to encode document positional information to assist models in capturing cross-document relations. However, existing MDS models, such as Transformer-based models, only consider token-level positional information. Moreover, these models fail to capture sentences' linguistic structure, which inevitably causes confusions in the generated summaries. Therefore, in this paper, we propose document-aware positional encoding and linguistic-guided encoding that can be fused with Transformer architecture for MDS. For document-aware positional encoding, we introduce a general protocol to guide the selection of document encoding functions. For linguistic-guided encoding, we propose to embed syntactic dependency relations into the dependency relation mask with a simple but effective non-linear encoding learner for feature learning. Extensive experiments show the proposed model can generate summaries with high quality.

Related papers

• Unified Multi-Modal Interleaved Document Representation for Information Retrieval [57.65409208879344]
  We produce more comprehensive and nuanced document representations by holistically embedding documents interleaved with different modalities. Specifically, we achieve this by leveraging the capability of recent vision-language models that enable the processing and integration of text, images, and tables into a unified format and representation.
  arXiv  Detail & Related papers  (2024-10-03T17:49:09Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization [1.066048003460524]
  Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. Abstractive MDS aims to generate a coherent and fluent summary for multiple documents using natural language generation techniques. We propose Absformer, a new Transformer-based method for unsupervised abstractive summary generation.
  arXiv  Detail & Related papers  (2023-06-07T21:18:23Z)
• Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering [49.85790367128085]
  We pre-training a generic multi-document model from a novel cross-document question answering pre-training objective. This novel multi-document QA formulation directs the model to better recover cross-text informational relations. Unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation and long text generation.
  arXiv  Detail & Related papers  (2023-05-24T17:48:40Z)
• A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization [66.08074487429477]
  Pre-trained language models (PLMs) have achieved outstanding achievements in abstractive single-document summarization (SDS) We propose a new method to better utilize a PLM to facilitate multi-document interactions for the multi-document summarization (MDS) task. Our method outperforms its corresponding PLM backbone by up to 3 Rouge-L and is favored by humans.
  arXiv  Detail & Related papers  (2023-05-15T10:03:31Z)
• A General-Purpose Multilingual Document Encoder [9.868221447090855]
  We pretrain a massively multilingual document encoder as a hierarchical transformer model (HMDE) We leverage Wikipedia as a readily available source of comparable documents for creating training data. We evaluate the effectiveness of HMDE in two arguably most common and prominent cross-lingual document-level tasks.
  arXiv  Detail & Related papers  (2023-05-11T17:55:45Z)
• Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval [79.37614949970013]
  We propose a new dense retrieval model which learns diverse document representations with deep query interactions. Our model encodes each document with a set of generated pseudo-queries to get query-informed, multi-view document representations.
  arXiv  Detail & Related papers  (2022-08-08T16:00:55Z)
• Unified Pretraining Framework for Document Understanding [52.224359498792836]
  We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses.
  arXiv  Detail & Related papers  (2022-04-22T21:47:04Z)
• Cross-Document Language Modeling [28.34202232940097]
  Cross-document language model (CD-LM) improves masked language modeling for multi-document NLP tasks. We show that our CD-LM sets new state-of-the-art results for several multi-text tasks.
  arXiv  Detail & Related papers  (2021-01-02T09:01:39Z)
• From Bag of Sentences to Document: Distantly Supervised Relation Extraction via Machine Reading Comprehension [22.39362905658063]
  We propose a new DS paradigm--document-based distant supervision. We show that our method achieves new state-of-the-art DS performance.
  arXiv  Detail & Related papers  (2020-12-08T10:16:27Z)
• Keyphrase Generation with Cross-Document Attention [28.565813544820553]
  Keyphrase generation aims to produce a set of phrases summarizing the essentials of a given document. We propose CDKGen, a Transformer-based keyphrase generator, which expands the Transformer to global attention. We also adopt a copy mechanism to enhance our model via selecting appropriate words from documents to deal with out-of-vocabulary words in keyphrases.
  arXiv  Detail & Related papers  (2020-04-21T07:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.