Tackling Query-Focused Summarization as A Knowledge-Intensive Task: A Pilot Study

• URL: http://arxiv.org/abs/2112.07536v2
• Date: Mon, 31 Jul 2023 12:10:22 GMT
• Title: Tackling Query-Focused Summarization as A Knowledge-Intensive Task: A Pilot Study
• Abstract summary: We tackle the Query-focused summarization (QFS) task as a knowledge-intensive (KI) task without access to any relevant documents. We build a new dataset (KI-QFS) by adapting existing QFS datasets. In this dataset, answering the query requires document retrieval from a knowledge corpus.
• Score: 24.453344448245726
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Query-focused summarization (QFS) requires generating a summary given a query using a set of relevant documents. However, such relevant documents should be annotated manually and thus are not readily available in realistic scenarios. To address this limitation, we tackle the QFS task as a knowledge-intensive (KI) task without access to any relevant documents. Instead, we assume that these documents are present in a large-scale knowledge corpus and should be retrieved first. To explore this new setting, we build a new dataset (KI-QFS) by adapting existing QFS datasets. In this dataset, answering the query requires document retrieval from a knowledge corpus. We construct three different knowledge corpora, and we further provide relevance annotations to enable retrieval evaluation. Finally, we benchmark the dataset with state-of-the-art QFS models and retrieval-enhanced models. The experimental results demonstrate that QFS models perform significantly worse on KI-QFS compared to the original QFS task, indicating that the knowledge-intensive setting is much more challenging and offers substantial room for improvement. We believe that our investigation will inspire further research into addressing QFS in more realistic scenarios.

Related papers

• Query-Focused Event Summarization: A Dataset and Benchmark [5.408715293537859]
  A thematic corpus is a collection of semantically coherent documents that collectively describe different aspects of a shared thematic event.<n>We propose the Query-Focused Event Summarization (QFES) task and construct the QFESum dataset, which contains 8 thematic events, 16,684 documents, and 104 queries.<n> Experimental results on QFESum show that RAT and SHC consistently outperform the baselines, demonstrating their effectiveness for QFES.
  arXiv  Detail & Related papers  (2026-07-13T07:02:50Z)
• PeerQA: A Scientific Question Answering Dataset from Peer Reviews [51.95579001315713]
  We present PeerQA, a real-world, scientific, document-level Question Answering dataset. The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP. We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks.
  arXiv  Detail & Related papers  (2025-02-19T12:24:46Z)
• IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization [59.06663981902496]
  Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. We investigate two indispensable characteristics that the LLMs-based QFS models should be harnessed, Lengthy Document Summarization and Efficiently Fine-grained Query-LLM Alignment. These innovations pave the way for broader application and accessibility in the field of QFS technology.
  arXiv  Detail & Related papers  (2024-07-15T07:14:56Z)
• DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs [3.24692739098077]
  Open-domain complex Question Answering (QA) is a difficult task with challenges in evidence retrieval and reasoning. We evaluate state-of-the-art pre-trained dense and sparse retrieval models in an open-domain setting. We observe that late interaction models and surprisingly lexical models like BM25 perform well compared to other pre-trained dense retrieval models.
  arXiv  Detail & Related papers  (2024-06-24T22:09:50Z)
• Automatic Question-Answer Generation for Long-Tail Knowledge [65.11554185687258]
  We propose an automatic approach to generate specialized QA datasets for tail entities. We conduct extensive experiments by employing pretrained LLMs on our newly generated long-tail QA datasets.
  arXiv  Detail & Related papers  (2024-03-03T03:06:31Z)
• QontSum: On Contrasting Salient Content for Query-focused Summarization [22.738731393540633]
  Query-focused summarization (QFS) is a challenging task in natural language processing that generates summaries to address specific queries. This paper highlights the role of QFS in Grounded Answer Generation (GAR) We propose QontSum, a novel approach for QFS that leverages contrastive learning to help the model attend to the most relevant regions of the input document.
  arXiv  Detail & Related papers  (2023-07-14T19:25:35Z)
• LMGQS: A Large-scale Dataset for Query-focused Summarization [77.6179359525065]
  We convert four generic summarization benchmarks into a new QFS benchmark dataset, LMGQS. We establish baselines with state-of-the-art summarization models. We achieve state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.
  arXiv  Detail & Related papers  (2023-05-22T14:53:45Z)
• Long-Tailed Question Answering in an Open World [46.67715607552547]
  We define Open Long-Tailed QA (OLTQA) as learning from long-tailed distributed data. We propose an OLTQA model that encourages knowledge sharing between head, tail and unseen tasks. On a large-scale OLTQA dataset, our model consistently outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2023-05-11T04:28:58Z)
• A Lightweight Constrained Generation Alternative for Query-focused Summarization [8.264410236351111]
  Query-focused summarization (QFS) aims to provide a summary of a document that satisfies information need of a given query. We propose leveraging a recently developed constrained generation model Neurological Decoding (NLD) as an alternative to current QFS regimes. We demonstrate the efficacy of this approach on two public QFS collections achieving near parity with the state-of-the-art model with substantially reduced complexity.
  arXiv  Detail & Related papers  (2023-04-23T18:43:48Z)
• Detect, Retrieve, Comprehend: A Flexible Framework for Zero-Shot Document-Level Question Answering [6.224211330728391]
  Researchers produce thousands of scholarly documents containing valuable technical knowledge. Document-level question answering (QA) offers a flexible framework where human-posed questions can be adapted to extract diverse knowledge. We present a three-stage document QA approach: text extraction from PDF; evidence retrieval from extracted texts to form well-posed contexts; and QA to extract knowledge from contexts to return high-quality answers.
  arXiv  Detail & Related papers  (2022-10-04T23:33:52Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• KILT: a Benchmark for Knowledge Intensive Language Tasks [102.33046195554886]
  We present a benchmark for knowledge-intensive language tasks (KILT) All tasks in KILT are grounded in the same snapshot of Wikipedia. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline.
  arXiv  Detail & Related papers  (2020-09-04T15:32:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.