Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models

• URL: http://arxiv.org/abs/2506.00773v2
• Date: Tue, 03 Jun 2025 13:29:29 GMT
• Title: Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
• Authors: Boheng Sheng, Jiacheng Yao, Meicong Zhang, Guoxiu He,
• Abstract summary: Large language models (LLMs) often struggle to accurately read and comprehend long texts.<n>Current methods for improvement typically rely on splitting long contexts into fixed-length chunks.<n>We propose a straightforward approach for dynamically separating and selecting chunks of long context.
• Score: 1.5817866616624976
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) often struggle to accurately read and comprehend extremely long texts. Current methods for improvement typically rely on splitting long contexts into fixed-length chunks. However, fixed truncation risks separating semantically relevant content, leading to ambiguity and compromising accurate understanding. To overcome this limitation, we propose a straightforward approach for dynamically separating and selecting chunks of long context, facilitating a more streamlined input for LLMs. In particular, we compute semantic similarities between adjacent sentences, using lower similarities to adaptively divide long contexts into variable-length chunks. We further train a question-aware classifier to select sensitive chunks that are critical for answering specific questions. Experimental results on both single-hop and multi-hop question-answering benchmarks show that the proposed approach consistently outperforms strong baselines. Notably, it maintains robustness across a wide range of input lengths, handling sequences of up to 256k tokens. Our datasets and code are available at the following link: https://github.com/ECNU-Text-Computing/DCS

Related papers

• Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs [23.960451986662996]
  This paper proposes a method that emulates Retrieval Augmented Generation (RAG) through specialized prompt engineering and chain-of-thought reasoning.<n>We evaluate our approach on selected tasks from BABILong, which interleaves standard bAbI QA problems with large amounts of distractor text.
  arXiv  Detail & Related papers  (2025-02-18T02:49:40Z)
• Squeezed Attention: Accelerating Long Context Length LLM Inference [64.11145320159126]
  We propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We use K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs.
  arXiv  Detail & Related papers  (2024-11-14T18:54:19Z)
• Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? [36.83397306207386]
  We evaluate the capabilities of 17 leading Large Language Models (LLMs)<n>Strikingly, many models are remarkably threadsafe: capable of simultaneously following multiple threads without significant loss in performance.<n>We find the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as the context window grows.
  arXiv  Detail & Related papers  (2024-11-07T18:59:27Z)
• Length-Induced Embedding Collapse in PLM-based Models [7.127156731612495]
  We introduce a phenomenon we call Length Collapse, where embeddings of longer texts tend to cluster together.<n>We investigate how these differences contribute to the performance decline observed with longer texts across various downstream tasks.<n>To address this issue, we propose a simple method, TempScale, which mitigates the Length Collapse phenomenon.
  arXiv  Detail & Related papers  (2024-10-31T17:55:36Z)
• What is Wrong with Perplexity for Long-context Language Modeling? [71.34933096461124]
  Long-context inputs are crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning.<n>Perplexity (PPL) has proven unreliable for assessing long-context capabilities.<n>We propose bfLongPPL, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them.
  arXiv  Detail & Related papers  (2024-10-31T09:39:28Z)
• Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception [10.614437503578856]
  This paper proposes the Meta-Chunking framework, which specifically enhances chunking quality.<n>We design two adaptive chunking techniques based on uncertainty, namely Perplexity Chunking and Margin Sampling Chunking.<n>We establish the global information compensation mechanism, encompassing a two-stage hierarchical summary generation process and a three-stage text chunk rewriting procedure.
  arXiv  Detail & Related papers  (2024-10-16T17:59:32Z)
• LLM$\ imes$MapReduce: Simplified Long-Sequence Processing using Large Language Models [73.13933847198395]
  We propose a training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding. The proposed LLM$times$MapReduce framework splits the entire document into several chunks for LLMs to read and then aggregates the intermediate answers to produce the final output.
  arXiv  Detail & Related papers  (2024-10-12T03:13:44Z)
• Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models [5.330795983408874]
  We introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks.
  arXiv  Detail & Related papers  (2024-09-07T03:54:46Z)
• M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models [58.54538318912159]
  M4LE is a benchmark for evaluating the long-sequence capability of large language models (LLMs) M4LE is based on a diverse NLP task pool comprising 36 NLP task types and 12 domains. We conducted a systematic evaluation on 11 well-established LLMs, especially those optimized for long-sequence inputs.
  arXiv  Detail & Related papers  (2023-10-30T03:11:30Z)
• Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading [63.93888816206071]
  We introduce MemWalker, a method that processes the long context into a tree of summary nodes. Upon receiving a query, the model navigates this tree in search of relevant information, and responds once it gathers sufficient information. We show that, beyond effective reading, MemWalker enhances explainability by highlighting the reasoning steps as it interactively reads the text; pinpointing the relevant text segments related to the query.
  arXiv  Detail & Related papers  (2023-10-08T06:18:14Z)
• Optimizing Factual Accuracy in Text Generation through Dynamic Knowledge Selection [71.20871905457174]
  Language models (LMs) have revolutionized the way we interact with information, but they often generate nonfactual text. Previous methods use external knowledge as references for text generation to enhance factuality but often struggle with the knowledge mix-up of irrelevant references. We present DKGen, which divide the text generation process into an iterative process.
  arXiv  Detail & Related papers  (2023-08-30T02:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.