Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs

• URL: http://arxiv.org/abs/2602.14784v1
• Date: Mon, 16 Feb 2026 14:32:18 GMT
• Title: Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs
• Authors: Christos Koutsiaris,
• Abstract summary: Intent-Driven Dynamic Chunking (IDC) is a novel approach that uses predicted user queries to guide document segmentation.<n>We evaluate IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.

Related papers

• Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval [37.055995647350784]
  This paper reproduces prior studies in document chunking and presents a systematic framework that unifies existing strategies.<n>Our evaluation reveals that optimal chunking strategies are task-dependent.
  arXiv  Detail & Related papers  (2026-02-19T00:27:15Z)
• Query Decomposition for RAG: Balancing Exploration-Exploitation [83.79639293409802]
  RAG systems address complex user requests by decomposing them into subqueries, retrieving potentially relevant documents for each, and then aggregating them to generate an answer.<n>We formulate query decomposition and document retrieval in an exploitation-exploration setting, where retrieving one document at a time builds a belief about the utility of a given sub-queries.<n>Our main finding is that estimating document relevance using rank information and human judgments yields a 35% gain in document-level precision, 15% increase in alpha-nDCG, and better performance on the downstream task of long-form generation.
  arXiv  Detail & Related papers  (2025-10-21T13:37:11Z)
• Chunk Knowledge Generation Model for Enhanced Information Retrieval: A Multi-task Learning Approach [13.945285357933487]
  This study proposes a method that divides documents into chunk units and generates textual data for each chunk to simultaneously improve retrieval efficiency and accuracy.<n>The proposed "Chunk Knowledge Generation Model" adopts a T5-based multi-task learning structure that simultaneously generates titles and candidate questions from each document chunk.<n> GPT-based evaluation on 305 query-document pairs showed that retrieval using the proposed model achieved 95.41% accuracy at Top@10.
  arXiv  Detail & Related papers  (2025-09-19T06:32:30Z)
• ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links [57.514511353084565]
  We introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links.<n>We apply our framework in two distinct domains -- peer review and news.<n>The resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review.
  arXiv  Detail & Related papers  (2025-09-01T11:32:24Z)
• Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking [58.69615583599489]
  Deliberate Thinking based Retriever (Debater) is a novel approach that enhances document representations by incorporating a step-by-step thinking process.<n>Debater significantly outperforms existing methods across several retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-02-18T15:56:34Z)
• Unveiling Document Structures with YOLOv5 Layout Detection [0.0]
  This research investigates the utilization of YOLOv5, a cutting-edge computer vision model, for the purpose of rapidly identifying document layouts and extracting unstructured data. The main objective is to create an autonomous system that can effectively recognize document layouts and extract unstructured data.
  arXiv  Detail & Related papers  (2023-09-29T07:45:10Z)
• Continual Learning for Generative Retrieval over Dynamic Corpora [115.79012933205756]
  Generative retrieval (GR) directly predicts the identifiers of relevant documents (i.e., docids) based on a parametric model.<n>The ability to incrementally index new documents while preserving the ability to answer queries is vital to applying GR models.<n>We put forward a novel Continual-LEarner for generatiVE Retrieval (CLEVER) model and make two major contributions to continual learning for GR.
  arXiv  Detail & Related papers  (2023-08-29T01:46:06Z)
• Generate rather than Retrieve: Large Language Models are Strong Context Generators [74.87021992611672]
  We present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
  arXiv  Detail & Related papers  (2022-09-21T01:30:59Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.