A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability

• URL: http://arxiv.org/abs/2608.00705v1
• Date: Sat, 01 Aug 2026 15:05:45 GMT
• Title: A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability
• Abstract summary: Reported verdicts on GraphRAG versus vector RAG disagree, and the evidence is typically tied to a single corpus, embedder, and judge.<n>We present a triple-robustness analysis that holds a five-pipeline architecture matrix fixed and varies embedder.<n>We argue that RAG architecture claims should be tested at this level of robustness before they are trusted.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reported verdicts on GraphRAG versus vector RAG disagree, and the evidence is typically tied to a single corpus, embedder, and judge -- and, we show, to where citation quality is measured. We present a triple-robustness analysis that holds a five-pipeline architecture matrix fixed and varies embedder (local e5-small vs. Azure text-embedding-3-small), corpus (DO-178C typed-edge requirements vs. Wikipedia paragraph chains via MuSiQue), and judge (paired GPT-5.4 x GPT-4.1 on both corpora), over 2x4,440 main-matrix runs, 600 cross-corpus runs, and over 5,000 faithfulness judgments. (C2a) GraphRAG's graph walk floods the context window at precision 0.12-0.23, but the synthesizer cites selectively at precision 0.48-0.65; scoring the retrieved set as the attribution set inverts the architecture ranking, which reconciles part of the disagreement in prior reports. (C1) Answer-level citation winners are corpus- and stratum-conditional but embedder-robust: GraphRAG ties vanilla on short-hop DO-178C queries and wins every MuSiQue stratum, while agentic pipelines lead only on 3+-hop requirements queries. (C2b) Faithfulness is corpus-conditional: on DO-178C it declines with hop distance (trend p<0.05 in three of four judge x embedder combinations); on Wikipedia chains neither judge shows a collapse. (C3) Single-judge LLM faithfulness is fragile to retrieval state: GPT-5.4's self-kappa across embedders is 0.137 (41% verdict change) against a same-day test-retest floor of 0.76, and re-judging frozen inputs eleven weeks later gives kappa <= 0.14 for both judges. A learned router on dense embeddings alone reaches macro-F1 0.86 on hop classification (C4). We argue that RAG architecture claims should be tested at this level of robustness -- including robustness to the citation-measurement point -- before they are trusted.

Related papers

• The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models [22.461983628330277]
  Many few-shot adaptation methods classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features.<n>We ask what is the right ratio, can it be estimated without validation data, and is finding it where the performance is?<n>Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity.<n> validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average.
  arXiv  Detail & Related papers  (2026-08-23T15:02:33Z)
• Structure-Aware Semantic Chunking with Title-Chain Prefixes: A 1600-Query Evaluation and the Measurement Trap in Text-Transform Ablations [4.342406076796542]
  We present a three-stage, chunk-side-only semantic chunking pipeline.<n>The title chain reuses the document's own header hierarchy instead of a generated summary.<n>The pipeline improves MRR@5 from 0.374 to 0.463 on the full set and from 0.828 to 0.925 on the answerable subset.
  arXiv  Detail & Related papers  (2026-08-01T19:01:46Z)
• InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy [0.9924185669370708]
  InvestPhilBench is a dynamic benchmark spanning eight cognitive tiers.<n>The v0.6 release comprises 118 primary-source-verified investment principle cards.<n>For reproducible scoring at scale we introduce the Benchmark Automated Scoring Pipeline.
  arXiv  Detail & Related papers  (2026-06-24T15:53:20Z)
• Decision-Aware Memory Cards: Counterfactual-Inspired Context Selection and Compression for Tool-Using LLM Agents [3.964533007623828]
  Modern large language model (LLM) agents need decision-relevant evidence at the moment of action.<n>We present the Counterfactual-Inspired Context Layer (CICL), which builds an instance context graph, estimates decision-oriented utility for candidate units, and compresses selected evidence into typed memory cards.<n>CICL provides a practical layer for measuring, ranking, and compressing decision-critical context for tool-using agents.
  arXiv  Detail & Related papers  (2026-06-06T13:02:28Z)
• Hierarchical Certified Semantic Commitment for Byzantine-Resilient LLM-Agent Collaboration [30.310793549183117]
  We introduce Hierarchical Certified Semantic Commitment (H-CSC), a BFT-inspired protocol that converts embedding-derived finality signals over verdict-emitting proposal groups into one of three typed outcomes.<n>H-CSC commits with low angular deviation on BFT-feasible buckets (0.31 to 2.04 degrees) and aborts 100% of beyond-BFT rounds (n3f+1) as intended.
  arXiv  Detail & Related papers  (2026-06-05T14:35:58Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks [41.99844472131922]
  We provide a benchmark calibration for foundation-model weak labels.<n>The crossover predicted by theory appears at ng100 on PCAM, 20-50 on ISIC, and 250-500 on NIH-CXR.<n>A structured-vs-random noise sign flip on NIH-CXR shows that the rate-only formulation of the bound is incomplete.
  arXiv  Detail & Related papers  (2026-05-23T23:15:07Z)
• Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation [24.67838359287715]
  We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings.<n>We collect 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings.<n>Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth.
  arXiv  Detail & Related papers  (2026-05-19T12:44:01Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Controlling Authority Retrieval: A Missing Retrieval Objective for Authority-Governed Knowledge [0.0]
  We call this CAR: retrieving the currently active authority for a semantic anchor q, that is, front(cl(A_k(q))).<n>We evaluate three real-world datasets: security advisories, SCOTUS overruling pairs, and FDA drug records.<n>A GPT-4omini experiment shows RAG produces explicit "not patched" claims for 39% of queries where a patch exists.
  arXiv  Detail & Related papers  (2026-04-15T23:56:35Z)
• OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection [0.0]
  We present a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and language models generate only surface prose.<n>The corpus spans 51 simulated days, 2,904 telemetry records at a 96.4% noise rate, and four detection scenarios designed to defeat single-surface and single-day triage strategies.
  arXiv  Detail & Related papers  (2026-03-23T19:03:53Z)
• QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs [29.26861081722613]
  We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics.<n>We introduce QEDBench, the first large-scale dual-rubric alignment benchmark to measure alignment with human experts on university-level math.<n>We reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias.
  arXiv  Detail & Related papers  (2026-02-24T07:23:28Z)
• BiRQA: Bidirectional Robust Quality Assessment for Images [49.74447451098852]
  Full-Reference image quality assessment (FR IQA) is important for image compression, restoration and generative modeling.<n>We present BiRQA, a compact FR IQA metric model that processes four fast complementary features within a bidirectional multiscale pyramid.<n>On five public FR IQA benchmarks BiRQA outperforms or matches the previous state of the art (SOTA) while running 3x faster than previous SOTA models.
  arXiv  Detail & Related papers  (2026-02-23T20:52:56Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  We present exMT, a benchmark for objective extraction and metacognition.<n>Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence.<n> Accuracy is scored by similarity to gold objectives, then thresholded once on 300 calibration items.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.