Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks

• URL: http://arxiv.org/abs/2605.09611v1
• Date: Sun, 10 May 2026 15:48:38 GMT
• Title: Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks
• Abstract summary: Preprint presents an empirical analysis of byte-exact chunk-level deduplication in Retrieval-Augmented Generation (RAG) pipelines.<n>We measure context reduction across three distinct operating regimes: clean academic retrieval, constructed enterprisevendor patterns, and multi-turn conversational AI.<n>All four vendors clear the strict 5% Wilson 95% upper-bound MAT threshold in both the clean and high-redundancy RAG regimes.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This preprint presents an empirical analysis of byte-exact chunk-level deduplication in Retrieval-Augmented Generation (RAG) pipelines. We measure context reduction across three distinct operating regimes: clean academic retrieval (0.16% byte reduction on 22.2M BeIR passages), constructed enterprise patterns (24.03% reduction), and multi-turn conversational AI (80.34% reduction). To validate quality preservation, we conducted a cross-vendor 5-judge calibrated panel evaluation across four production APIs (Google Gemini 2.5 Flash, Anthropic Claude Sonnet 4.6, Meta Llama 3.3 70B, and OpenAI GPT-5.1). Applying a five-category human-in-the-loop noise-removal protocol to panel-majority materially different (MAT) pairs, we establish that byte-exact deduplication introduces zero measurable quality regression. Post-audit, all four vendors clear the strict <5% Wilson 95% upper-bound MAT threshold in both the clean and high-redundancy RAG regimes. This work demonstrates that substantial inference compute savings can be achieved deterministically without compromising evaluation-grade model quality.

Related papers

• Recursive Agentic Reasoning [4.7490116928645065]
  Test-time reasoning methods such as iterative refinement, decomposition, and repeated sampling are often evaluated in isolation.<n>We introduce a unified view of these methods as recursion operators over an agent's reasoning trace.<n>We evaluate all three operators against a single-pass chain-of-thought baseline under a shared harness with identical prompts, token budgets, and grading code.
  arXiv  Detail & Related papers  (2026-08-25T01:34:08Z)
• Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge [46.547776251310694]
  This paper evaluates 5G domain understanding and fault analysis in a free-text generation format.<n>We evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks.
  arXiv  Detail & Related papers  (2026-08-21T12:09:51Z)
• SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution [6.908637308550535]
  SEVA is a structured verification agent that emits evidence alignments, step-by-step reasoning chains, calibrated confidence, and a six-category error diagnosis.<n>On ClearFacts, SEVA-3B matches GPT-4o-mini (69.0 vs. 69.8 F1) while producing substantially richer, auditable output.
  arXiv  Detail & Related papers  (2026-06-29T02:37:13Z)
• An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection [0.0]
  We present COVA-X, an expanded dataset of 10,985 conversations spanning eight elder-targeted scam categories.<n>Longformer now surpasses XGBoost on all evaluation metrics, achieving 79.71% accuracy and 0.7786 macro F1 compared with 78.43% and 0.7563 for XGBoost.
  arXiv  Detail & Related papers  (2026-06-05T03:46:36Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Human-Inspired Memory Architecture for LLM Agents [0.9507070656654629]
  We present a biologically-grounded memory architecture comprising six cognitive mechanisms.<n>Each mechanism addresses a specific failure mode of naive memory accumulation.<n>At S-tier scale (50 sessions), dedup-based consolidation yields a +13.3 pp improvement in preference recall.
  arXiv  Detail & Related papers  (2026-05-08T22:52:37Z)
• Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels [0.0]
  Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment.<n>Existing studies typically compare only two conditions, rely on aggregate bias metrics, and evaluate a single model family.<n>We conduct a controlled empirical study of three instruction-tuned models at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items.
  arXiv  Detail & Related papers  (2026-05-02T05:41:47Z)
• GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs [0.0]
  We present a groundedness framework that partitions claims into a four-way typology (grounded, ungrounded, contradicted, complementary)<n>GSAR is the first published groundedness framework coupling evidence-typed scoring with tiered recovery under an explicit compute budget.
  arXiv  Detail & Related papers  (2026-04-25T16:20:28Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• RecGPT-V2 Technical Report [41.57739441038769]
  Large language models (LLMs) have demonstrated remarkable potential in transforming recommender systems from implicit behavioral pattern matching to explicit intent reasoning.<n>We present RecGPT-V2 with four key innovations to transform recommender systems from implicit behavioral pattern matching to explicit intent reasoning.<n>Online A/B tests on Taobao demonstrate significant improvements: +2.98% CTR, +3.71% IPV, +2.19% TV, and +11.46% NER.
  arXiv  Detail & Related papers  (2025-12-16T15:40:44Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data? [82.09573568241724]
  EssenceBench is a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA)<n>Our approach yields superior compression results with lower reconstruction error and markedly higher efficiency.<n>On the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples.
  arXiv  Detail & Related papers  (2025-10-12T05:38:10Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks [0.0]
  Confidence-diversity calibration is a quality assessment framework for accessible coding tasks.<n>Analysing 5,680 coding decisions from eight state-of-the-art LLMs, we find that mean self-confidence tracks inter-model agreement closely.
  arXiv  Detail & Related papers  (2025-08-04T03:47:10Z)
• MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning [80.15393178083607]
  Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples.<n> Refinement offers an alternative by using LLM-generated feedback to improve solution quality.<n>We propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard.
  arXiv  Detail & Related papers  (2024-09-18T17:12:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.