A Sentiment Consolidation Framework for Meta-Review Generation

• URL: http://arxiv.org/abs/2402.18005v2
• Date: Tue, 4 Jun 2024 16:10:13 GMT
• Title: A Sentiment Consolidation Framework for Meta-Review Generation
• Authors: Miao Li, Jey Han Lau, Eduard Hovy,
• Abstract summary: We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. We propose novel prompting methods for Large Language Models to generate meta-reviews.
• Score: 40.879419691373826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern natural language generation systems with Large Language Models (LLMs) exhibit the capability to generate a plausible summary of multiple documents; however, it is uncertain if they truly possess the capability of information consolidation to generate summaries, especially on documents with opinionated information. We focus on meta-review generation, a form of sentiment summarisation for the scientific domain. To make scientific sentiment summarization more grounded, we hypothesize that human meta-reviewers follow a three-layer framework of sentiment consolidation to write meta-reviews. Based on the framework, we propose novel prompting methods for LLMs to generate meta-reviews and evaluation metrics to assess the quality of generated meta-reviews. Our framework is validated empirically as we find that prompting LLMs based on the framework -- compared with prompting them with simple instructions -- generates better meta-reviews.

Related papers

• Meta Knowledge for Retrieval Augmented Large Language Models [0.0]
  We introduce a novel data-centric RAG workflow for Large Language Models (LLMs) Our methodology relies on generating metadata and synthetic Questions and Answers (QA) for each document. We demonstrate that using augmented queries with synthetic question matching significantly outperforms traditional RAG pipelines.
  arXiv  Detail & Related papers  (2024-08-16T20:55:21Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• Prompting LLMs to Compose Meta-Review Drafts from Peer-Review Narratives of Scholarly Manuscripts [6.2701471990853594]
  Large Language Models (LLMs) can generate meta-reviews based on peer-review narratives from multiple experts. In this paper, we perform a case study with three popular LLMs to automatically generate meta-reviews.
  arXiv  Detail & Related papers  (2024-02-23T20:14:16Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Knowledge-Centric Templatic Views of Documents [2.654058995940072]
  Authors often share their ideas in various document formats, such as slide decks, newsletters, reports, and posters. We introduce a novel unified evaluation framework that can be adapted to measuring the quality of document generators. We conduct a human evaluation, which shows that people prefer 82% of the documents generated with our method.
  arXiv  Detail & Related papers  (2024-01-13T01:22:15Z)
• Absformer: Transformer-based Model for Unsupervised Multi-Document Abstractive Summarization [1.066048003460524]
  Multi-document summarization (MDS) refers to the task of summarizing the text in multiple documents into a concise summary. Abstractive MDS aims to generate a coherent and fluent summary for multiple documents using natural language generation techniques. We propose Absformer, a new Transformer-based method for unsupervised abstractive summary generation.
  arXiv  Detail & Related papers  (2023-06-07T21:18:23Z)
• Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation [45.9443710073576]
  We present PeerSum, a novel dataset for generating meta-reviewes of scientific papers. Rammer is a model that uses sparse attention based on the conversational structure and a training objective that predicts metadata features.
  arXiv  Detail & Related papers  (2023-05-02T15:18:18Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.