Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation

• URL: http://arxiv.org/abs/2305.01498v4
• Date: Mon, 23 Oct 2023 06:18:09 GMT
• Title: Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation
• Authors: Miao Li, Eduard Hovy, Jey Han Lau
• Abstract summary: We present PeerSum, a novel dataset for generating meta-reviewes of scientific papers. Rammer is a model that uses sparse attention based on the conversational structure and a training objective that predicts metadata features.
• Score: 45.9443710073576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present PeerSum, a novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as abstractive summaries of reviews, multi-turn discussions and the paper abstract. These source documents have rich inter-document relationships with an explicit hierarchical conversational structure, cross-references and (occasionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce Rammer ( Relationship-aware Multi-task Meta-review Generator), a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features (e.g., review ratings). Our experimental results show that Rammer outperforms other strong baseline models in terms of a suite of automatic evaluation metrics. Further analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents of PeerSum, suggesting meta-review generation is a challenging task and a promising avenue for further research.

Related papers

• Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction [23.47150047875133]
  Document parsing is essential for converting unstructured and semi-structured documents into machine-readable data. Document parsing plays an indispensable role in both knowledge base construction and training data generation. This paper discusses the challenges faced by modular document parsing systems and vision-language models in handling complex layouts.
  arXiv  Detail & Related papers  (2024-10-28T16:11:35Z)
• Interactive Topic Models with Optimal Transport [75.26555710661908]
  We present EdTM, as an approach for label name supervised topic modeling. EdTM models topic modeling as an assignment problem while leveraging LM/LLM based document-topic affinities.
  arXiv  Detail & Related papers  (2024-06-28T13:57:27Z)
• Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions [62.0123588983514]
  Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields. We reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources.
  arXiv  Detail & Related papers  (2024-06-09T08:24:17Z)
• A Sentiment Consolidation Framework for Meta-Review Generation [40.879419691373826]
  We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. We propose novel prompting methods for Large Language Models to generate meta-reviews.
  arXiv  Detail & Related papers  (2024-02-28T02:40:09Z)
• Knowledge-Centric Templatic Views of Documents [2.654058995940072]
  Authors often share their ideas in various document formats, such as slide decks, newsletters, reports, and posters. We introduce a novel unified evaluation framework that can be adapted to measuring the quality of document generators. We conduct a human evaluation, which shows that people prefer 82% of the documents generated with our method.
  arXiv  Detail & Related papers  (2024-01-13T01:22:15Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• Topic-Guided Abstractive Multi-Document Summarization [21.856615677793243]
  A critical point of multi-document summarization (MDS) is to learn the relations among various documents. We propose a novel abstractive MDS model, in which we represent multiple documents as a heterogeneous graph. We employ a neural topic model to jointly discover latent topics that can act as cross-document semantic units.
  arXiv  Detail & Related papers  (2021-10-21T15:32:30Z)
• Aspect-Controllable Opinion Summarization [58.5308638148329]
  We propose an approach that allows the generation of customized summaries based on aspect queries. Using a review corpus, we create a synthetic training dataset of (review, summary) pairs enriched with aspect controllers. We fine-tune a pretrained model using our synthetic dataset and generate aspect-specific summaries by modifying the aspect controllers.
  arXiv  Detail & Related papers  (2021-09-07T16:09:17Z)
• Automatic Document Sketching: Generating Drafts from Analogous Texts [44.626645471195495]
  We introduce a new task, document sketching, which involves generating entire draft documents for the writer to review and revise. These drafts are built from sets of documents that overlap in form - sharing large segments of potentially reusable text - while diverging in content. We investigate the application of weakly supervised methods, including use of a transformer-based mixture of experts, together with reinforcement learning.
  arXiv  Detail & Related papers  (2021-06-14T06:46:06Z)
• ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining [61.82562838486632]
  We crowdsource four new datasets on diverse online conversation forms of news comments, discussion forums, community question answering forums, and email threads. We benchmark state-of-the-art models on our datasets and analyze characteristics associated with the data.
  arXiv  Detail & Related papers  (2021-06-01T22:17:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.