Structure-Aware Semantic Chunking with Title-Chain Prefixes: A 1600-Query Evaluation and the Measurement Trap in Text-Transform Ablations

• URL: http://arxiv.org/abs/2608.00824v1
• Date: Sat, 01 Aug 2026 19:01:46 GMT
• Title: Structure-Aware Semantic Chunking with Title-Chain Prefixes: A 1600-Query Evaluation and the Measurement Trap in Text-Transform Ablations
• Abstract summary: We present a three-stage, chunk-side-only semantic chunking pipeline.<n>The title chain reuses the document's own header hierarchy instead of a generated summary.<n>The pipeline improves MRR@5 from 0.374 to 0.463 on the full set and from 0.828 to 0.925 on the answerable subset.
• Score: 4.342406076796542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chunking is the first and most consequential step in retrieval-augmented generation (RAG): every downstream retrieval decision inherits the chunk boundaries. We present a three-stage, chunk-side-only semantic chunking pipeline---header-split, semantic merge, and title-chain prefixing---that costs zero additional LLM calls: the title chain reuses the document's own header hierarchy instead of a generated summary. On a 1600-query stratified evaluation over a production Markdown knowledge base, the pipeline improves MRR@5 from 0.374 to 0.463 (+23.8%) on the full set and from 0.828 to 0.925 (+11.7%) on the answerable subset (n=563), with dual-annotator Cohen's kappa 0.45 (unweighted, 16,000 score pairs). We then report what we tried and what failed: three query-side or architecture-level follow-ups are design dead ends (unevaluated---no comparable run artifacts), one measured failure (prefix weight decay), and one protocol-level failure that is the paper's central methodological finding. In a same-pool prefix on/off ablation, dual-annotator agreement collapsed from kappa 0.45 to 0.04 under identical prompts---stripping the title-chain context strips the disambiguation signal annotators need to agree on relevance. This measurement trap invalidates a common evaluation practice in chunking research and motivates retrieval-time per-candidate prefix evaluation, the direction we recommend from all our evidence.

Related papers

• Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows [0.0]
  Compound AI pipelines chain LLM calls, retrievers, and tools are nondeterministic.<n>We give a protocol covering all three: it commits digests of each stage's inputs, outputs, and context.<n>On a synthetic HotpotQA pipeline a calibrated fixed threshold accepts 44 of 45 honest reproductions and rejects 104 of 105 divergent pairs.
  arXiv  Detail & Related papers  (2026-09-07T22:10:54Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study [32.505127447635864]
  We study controlled top-1 recognition over Schwartz's ten basic values.<n>We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol.<n>We find that models often locate the correct motivational region while ranking close alternatives unstably.
  arXiv  Detail & Related papers  (2026-07-22T15:22:14Z)
• The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling [8.616356693448987]
  We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays.<n>CCB fixes the semantic content of a task and varies only its depth N in 5,...,50 across three structurally distinct regimes.<n>We find a consistent pattern of geometric per-step decay with widely separated domain ceilings.
  arXiv  Detail & Related papers  (2026-06-28T08:56:34Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Hierarchical Certified Semantic Commitment for Byzantine-Resilient LLM-Agent Collaboration [30.310793549183117]
  We introduce Hierarchical Certified Semantic Commitment (H-CSC), a BFT-inspired protocol that converts embedding-derived finality signals over verdict-emitting proposal groups into one of three typed outcomes.<n>H-CSC commits with low angular deviation on BFT-feasible buckets (0.31 to 2.04 degrees) and aborts 100% of beyond-BFT rounds (n3f+1) as intended.
  arXiv  Detail & Related papers  (2026-06-05T14:35:58Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites: ML Classifiers and LLM-Judge Baselines [1.9537983097153042]
  Behaviour-Driven Development (BDD) software test suites accumulate duplicated step subsequences.<n>Three published patterns are available (within-file Background, within-repo reusable-scenario invocation, cross-organisational shared higher-level step)<n>No prior work automates which recurring subsequences are worth extracting or which mechanism applies.
  arXiv  Detail & Related papers  (2026-05-14T08:38:04Z)
• Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark [1.9537983097153042]
  Behaviour-Driven Development suites accumulate step-duplicate duplication with documented maintenance cost.<n>We release the largest cross-organisational BDD step corpus to date.
  arXiv  Detail & Related papers  (2026-04-22T11:44:05Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• Evidence Units: Ontology-Grounded Document Organization for Parser-Independent Retrieval [0.0]
  Element-level indexing treats every element parsed as an independent chunk, scattering semantically cohesive units across separate retrieval candidates.<n>This paper presents a pipeline that constructs Evidence Units (EUs): semantically complete document chunks that group visual assets with their contextual text.
  arXiv  Detail & Related papers  (2026-04-01T05:32:16Z)
• LLM-guided Hierarchical Retrieval [54.73080745446999]
  LATTICE is a hierarchical retrieval framework that enables an LLM to reason over and navigate large corpora with logarithmic search complexity.<n>A central challenge in such LLM-guided search is that the model's relevance judgments are noisy, context-dependent, and unaware of the hierarchy.<n>Our framework achieves state-of-the-art zero-shot performance on the reasoning-intensive BRIGHT benchmark.
  arXiv  Detail & Related papers  (2025-10-15T07:05:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.