Parallel Hierarchical Transformer with Attention Alignment for Abstractive Multi-Document Summarization

• URL: http://arxiv.org/abs/2208.07845v1
• Date: Tue, 16 Aug 2022 17:02:48 GMT
• Title: Parallel Hierarchical Transformer with Attention Alignment for Abstractive Multi-Document Summarization
• Authors: Ye Ma and Lu Zong
• Abstract summary: Abstractive Multi-Document Summarization (MDS) brings challenges on the representation and coverage of its lengthy and linked sources. This study develops a Parallel Hierarchical Transformer (PHT) with attention alignment for MDS.
• Score: 4.035753155957699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In comparison to single-document summarization, abstractive Multi-Document Summarization (MDS) brings challenges on the representation and coverage of its lengthy and linked sources. This study develops a Parallel Hierarchical Transformer (PHT) with attention alignment for MDS. By incorporating word- and paragraph-level multi-head attentions, the hierarchical architecture of PHT allows better processing of dependencies at both token and document levels. To guide the decoding towards a better coverage of the source documents, the attention-alignment mechanism is then introduced to calibrate beam search with predicted optimal attention distributions. Based on the WikiSum data, a comprehensive evaluation is conducted to test improvements on MDS by the proposed architecture. By better handling the inner- and cross-document information, results in both ROUGE and human evaluation suggest that our hierarchical model generates summaries of higher quality relative to other Transformer-based baselines at relatively low computational cost.

Related papers

• HDT: Hierarchical Document Transformer [70.2271469410557]
  HDT exploits document structure by introducing auxiliary anchor tokens and redesigning the attention mechanism into a sparse multi-level hierarchy. We develop a novel sparse attention kernel that considers the hierarchical structure of documents.
  arXiv  Detail & Related papers  (2024-07-11T09:28:04Z)
• Unsupervised Multi-document Summarization with Holistic Inference [41.58777650517525]
  This paper proposes a new holistic framework for unsupervised multi-document extractive summarization. Subset Representative Index (SRI) balances the importance and diversity of a subset of sentences from the source documents. Our findings suggest that diversity is essential for improving multi-document summary performance.
  arXiv  Detail & Related papers  (2023-09-08T02:56:30Z)
• Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization [1.066048003460524]
  Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. Abstractive MDS aims to generate a coherent and fluent summary for multiple documents using natural language generation techniques. We propose Absformer, a new Transformer-based method for unsupervised abstractive summary generation.
  arXiv  Detail & Related papers  (2023-06-07T21:18:23Z)
• Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods [4.211128681972148]
  Topic-controllable summarization is an emerging research area with a wide range of potential applications. This work proposes a new topic-oriented evaluation measure to automatically evaluate the generated summaries. In addition, we adapt topic embeddings to work with powerful Transformer architectures and propose a novel and efficient approach for guiding the summary generation through control tokens.
  arXiv  Detail & Related papers  (2022-06-09T07:28:16Z)
• ACM -- Attribute Conditioning for Abstractive Multi Document Summarization [0.0]
  We propose a model that incorporates attribute conditioning modules in order to decouple conflicting information by conditioning for a certain attribute in the output summary. This approach shows strong gains in ROUGE score over baseline multi document summarization approaches.
  arXiv  Detail & Related papers  (2022-05-09T00:00:14Z)
• Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
  We propose a principled inference framework to improve summarization models on two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
  arXiv  Detail & Related papers  (2022-03-15T01:24:51Z)
• Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution [48.093500219958834]
  We propose an Accelerated Multi-Scale Aggregation network (AMSA) for Reference-based Super-Resolution. The proposed AMSA achieves superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2022-01-12T08:40:23Z)
• PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policies [68.8204255655161]
  We propose a reinforcement learning based framework PoBRL for solving multi-document summarization. Our strategy decouples this multi-objective optimization into different subproblems that can be solved individually by reinforcement learning. Our empirical analysis shows state-of-the-art performance on several multi-document datasets.
  arXiv  Detail & Related papers  (2021-05-18T02:55:42Z)
• Data Augmentation for Abstractive Query-Focused Multi-Document Summarization [129.96147867496205]
  We present two QMDS training datasets, which we construct using two data augmentation methods. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. We build end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets.
  arXiv  Detail & Related papers  (2021-03-02T16:57:01Z)
• Document-level Neural Machine Translation with Document Embeddings [82.4684444847092]
  This work focuses on exploiting detailed document-level context in terms of multiple forms of document embeddings. The proposed document-aware NMT is implemented to enhance the Transformer baseline by introducing both global and local document-level clues on the source end.
  arXiv  Detail & Related papers  (2020-09-16T19:43:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.