ChapterBreak: A Challenge Dataset for Long-Range Language Models

• URL: http://arxiv.org/abs/2204.10878v1
• Date: Fri, 22 Apr 2022 18:20:23 GMT
• Title: ChapterBreak: A Challenge Dataset for Long-Range Language Models
• Authors: Simeng Sun, Katherine Thai, Mohit Iyyer
• Abstract summary: We introduce ChapterBreak, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions. Experiments on ChapterBreak show that existing LRLMs fail to effectively leverage long-range context.
• Score: 36.54750186213335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While numerous architectures for long-range language models (LRLMs) have recently been proposed, a meaningful evaluation of their discourse-level language understanding capabilities has not yet followed. To this end, we introduce ChapterBreak, a challenge dataset that provides an LRLM with a long segment from a narrative that ends at a chapter boundary and asks it to distinguish the beginning of the ground-truth next chapter from a set of negative segments sampled from the same narrative. A fine-grained human annotation reveals that our dataset contains many complex types of chapter transitions (e.g., parallel narratives, cliffhanger endings) that require processing global context to comprehend. Experiments on ChapterBreak show that existing LRLMs fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task. We publicly release our ChapterBreak dataset to spur more principled future research into LRLMs.

Related papers

• LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models [73.13933847198395]
  We propose a training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding. The proposed LLM$times$MapReduce framework splits the entire document into several chunks for LLMs to read and then aggregates the intermediate answers to produce the final output.
  arXiv  Detail & Related papers  (2024-10-12T03:13:44Z)
• A Novel LLM-based Two-stage Summarization Approach for Long Dialogues [9.835499880812646]
  This study proposes a hierarchical framework that segments and condenses information from long documents. The condensation stage utilizes an unsupervised generation model to generate condensed data. The summarization stage fine-tunes the abstractive summarization model on the condensed data to generate the final results.
  arXiv  Detail & Related papers  (2024-10-09T03:42:40Z)
• FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models [54.13671100638092]
  We propose a fragment-connected Hierarchical Memory based Large Language Models (LLMs) We formulate the fragment-level relations in external memory and present several instantiations for different text types. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.
  arXiv  Detail & Related papers  (2024-06-05T09:31:37Z)
• LaSagnA: Language-based Segmentation Assistant for Complex Queries [39.620806493454616]
  Large Language Models for Vision (vLLMs) generate detailed perceptual outcomes, including bounding boxes and masks. In this study, we acknowledge that the main cause of these problems is the insufficient complexity of training queries. We present three novel strategies to effectively handle the challenges arising from the direct integration of the proposed format.
  arXiv  Detail & Related papers  (2024-04-12T14:40:45Z)
• From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions [63.11097464396147]
  We introduce a novel benchmark YTSeg focusing on spoken content that is inherently more unstructured and both topically and structurally diverse. We also introduce an efficient hierarchical segmentation model MiniSeg, that outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-27T15:59:37Z)
• LISA: Reasoning Segmentation via Large Language Model [68.24075852136761]
  We propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. We present LISA: large Language Instructed Assistant, which inherits the language generation capabilities of multimodal Large Language Models.
  arXiv  Detail & Related papers  (2023-08-01T17:50:17Z)
• PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents [78.27865456183397]
  We propose PEARL, a prompting framework to improve reasoning over long documents. Each stage of PEARL is implemented via zero-shot or few-shot prompting with minimal human input. We evaluate PEARL on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts.
  arXiv  Detail & Related papers  (2023-05-23T23:06:04Z)
• Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents [13.755637074366813]
  SummN is a simple, flexible, and effective multi-stage framework for input texts longer than the maximum context lengths of typical pretrained LMs. It can process input text of arbitrary length by adjusting the number of stages while keeping the LM context size fixed. Our experiments demonstrate that SummN significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-16T06:19:54Z)
• BookSum: A Collection of Datasets for Long-form Narrative Summarization [42.26628743419607]
  BookSum is a collection of datasets for long-form narrative summarization. Our dataset covers source documents from the literature domain, such as novels, plays and stories.
  arXiv  Detail & Related papers  (2021-05-18T00:22:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.