Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

• URL: http://arxiv.org/abs/2608.22856v1
• Date: Mon, 24 Aug 2026 06:40:09 GMT
• Title: Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
• Abstract summary: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed.<n>We call the hidden phenomenon accuracy-blind answer churn and introduce the emphSnapshot Compatibility Audit, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement.
• Score: 0.4351760066527804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the \emph{Snapshot Compatibility Audit}, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only $-1.50$ points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

Related papers

• Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay [3.888537003098864]
  We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings.<n>We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system.<n>Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue.
  arXiv  Detail & Related papers  (2026-08-27T09:15:33Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• When Should Multi-Round RAG Stop? Structured Stopping Judgments and Retrieval Reduction in Search-R1 [0.0]
  Multi-round retrieval-augmented generation (RAG) must decide when to stop searching as evidence accumulates.<n>We adapt S2G-RAG's structured sufficiency-and-gap judgment to a frozen Search-R1 pipeline.<n>We train a Qwen3.5-2B judge on 3,009 states from 900 disjoint HotpotQA questions.
  arXiv  Detail & Related papers  (2026-08-13T13:42:48Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• CUSUM-Shaped Inference-Time Monitoring and Targeted Re-Decoding for Quantized Small Language Model Reasoning [0.6999740786886536]
  MGT-B maps overlapping windows of pre-sampling uncertainty and degeneration features to position-conditional empirical tail probabilities.<n>We audit whether the effect persists on problem identities first observed after the manual choice of log threshold h = 10.<n>Neither analysis is confirmatory, and the empirical factors are not established as a valid e-process or e-detector.
  arXiv  Detail & Related papers  (2026-07-22T13:34:17Z)
• Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation [10.60293484295824]
  We formulate adaptive RAG as calibrated retrieval-budget allocation.<n>We calibrate sequence log-probability and prefix-logit uncertainty signals into probabilities of correctness.<n>For sequence log-probability, ECE drops from 0.275 to 0.062 on TriviaQA, 0.643 to 0.009 on NQ, and 0.711 to 0.031 on MS MARCO.
  arXiv  Detail & Related papers  (2026-06-29T08:36:33Z)
• Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning [15.231928045776293]
  sevra is a serving-layer controller that decides whether to preserve a frozen solver's initial answer or invoke active verification.<n>On mathfive, selective verification reaches 76.3% accuracy, compared with 75.5% for always verifying.<n>On CommonsenseQA, always-on verification hurts, while Self-Consistency@5 improves accuracy about five times the realized token cost.
  arXiv  Detail & Related papers  (2026-06-18T05:25:43Z)
• FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search [88.16262636915975]
  FineVerify is a fine-grained self-verification framework for agentic search.<n>It decomposes each question into checkable sub-questions, verifies sampled candidates, and selects the candidate with the highest aggregated score.<n>FineVerify consistently outperforms standard scaling baselines.
  arXiv  Detail & Related papers  (2026-05-30T10:21:20Z)
• MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation [51.32912774464992]
  Large language models often solve tasks from a fully specified prompt but degrade when the same requirements unfold over multiple turns.<n>We propose MAIGO, an on-policy self-distillation method that reduces this contamination using history-cleaned references from the model's own policy.<n>For middle turns, MAIGO removes prior assistant replies while preserving the user-visible sharded prefix; for answer turns, it distills from paired full-view references conditioned on the completed user-side dialogue.
  arXiv  Detail & Related papers  (2026-05-26T15:38:46Z)
• PAR$^2$-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering [57.89576196160413]
  Large language models (LLMs) remain brittle on multi-hop question answering (MHQA)<n>We propose textbfPlanned Active Retrieval and Reasoning RAG (PAR$2$-RAG), a two-stage framework that separates emphcoverage from emphcommitment.
  arXiv  Detail & Related papers  (2026-03-30T23:52:54Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.