Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG

• URL: http://arxiv.org/abs/2605.28044v1
• Date: Wed, 27 May 2026 06:47:44 GMT
• Title: Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG
• Abstract summary: We introduce FORCEBENCH, a contrastive stress test for evidence-force calibration.<n>Each item holds a cited passage fixed and pairs an evidence-calibrated claim with a localized force-raised variant.<n>Headline experiments use a fixed, locality-filtered 198-pair evaluation set.
• Score: 6.113159481005088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cited RAG evaluation often treats visible sources as a grounding signal, but a real, topically relevant citation can still under-warrant the attached wording. We study this diagnostic failure as citation laundering: a related source is presented as warrant for an over-strong claim. We introduce FORCEBENCH, a contrastive stress test for evidence-force calibration. Each item holds a cited passage fixed and pairs an evidence-calibrated claim with a localized force-raised variant across five operational axes: relation, modality, scope, temporal validity, and numeric specificity. A calibrated evaluator should score the evidence-calibrated claim higher. Headline experiments use a fixed, locality-filtered 198-pair evaluation set. A citation-presence sanity check is uninformative by design; token and entity overlap still violate monotonicity on 32.8--36.4% of pairs. Across four reported model judges, standard generic support prompting is insufficient for this force-calibration stress test (aggregate MVR 47.2%), while explicit warrant-strength prompting lowers MVR to 24.5% but remains imperfect. We release the benchmark, prompts, outputs, and plug-in pipeline so citation evaluators can report monotonicity violation rate and force sensitivity alongside conventional support metrics.

Related papers

• Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation [8.796833451421493]
  LLM-as-a-judge protocols are commonly debiased by instructing judges to ignore presentation cues.<n>We show that this intervention can suppress bias while damaging the resolution of the measurement instrument.<n>We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation.
  arXiv  Detail & Related papers  (2026-09-11T04:59:10Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models [7.714704554667688]
  Large Language Models (LLMs) increasingly support decision-making in high-stakes domains.<n>LLMs often hallucinate and express confidence that is misaligned with factual correctness.<n>We study claim-level confidence calibration as a decision-relevant uncertainty signal.
  arXiv  Detail & Related papers  (2026-08-23T16:09:54Z)
• From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents [47.678562263387214]
  Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings.<n>Yet a completed trajectory is not automatically evidence: generated artifacts may be unsupported or incomplete, executed rounds may be invalid or confounded, and later modifications may obscure earlier findings.<n>We introduce an evidence-grounded framework that couples bounded verification of consequential artifacts with post-execution claim qualification.
  arXiv  Detail & Related papers  (2026-08-05T13:37:38Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric [0.0]
  No vendor-neutral, runnable instrument scores reconstructability as an evaluation-validity metric.<n>This paper introduces a property-level reconstructability metric over eight decision-property classes.
  arXiv  Detail & Related papers  (2026-07-14T07:54:37Z)
• The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking [1.918675896078337]
  We introduce SIFT -- claim-conditioned re-scoring of extracted evidence spans against the full claim.<n>We evaluate on FEVER, SciFact, 5PILS, and DP across four open-source backbones.
  arXiv  Detail & Related papers  (2026-06-23T14:23:56Z)
• Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction [4.273094752480624]
  We introduce prover-verifier deliberation (PVD), an inference-time protocol grounded in interactive proof theory.<n>PVD produces both an answer and a structured confidence verdict, allowing a system to report high-confidence answers while abstaining on uncertain cases.
  arXiv  Detail & Related papers  (2026-05-24T15:23:27Z)
• Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation [0.0]
  Mixed evidence may support one claim, require conditions for another, and contradict a third.<n>We study claimselective responses into verifiable claims, scored against retrieved evidence, and mapped by an intent-aware selector to full, partial, conflict, abstain<n>The resulting interface separates action-label prediction from evidence-linked claim selection under mixed evidence.
  arXiv  Detail & Related papers  (2026-05-21T03:29:50Z)
• Reward Hacking in Rubric-Based Reinforcement Learning [23.418394508756464]
  We study reward hacking in rubric-based RL, where a policy is optimized against a training verifier but evaluated against a cross-family panel of three frontier judges.<n>Our framework separates two sources of divergence: verifier failure and rubric-design limitations.<n>Across medical and science domains, weak verifiers produce large proxy-reward gains that do not transfer to the reference verifiers.
  arXiv  Detail & Related papers  (2026-05-12T17:54:25Z)
• Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone [11.663456969895462]
  Alignment evaluation in machine learning has largely become evaluation of models.<n>This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone.
  arXiv  Detail & Related papers  (2026-05-06T03:28:30Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal [31.458406135473805]
  We present UniCR, a unified framework that turns heterogeneous uncertainty evidence into a calibrated probability of correctness.<n>UniCR learns a lightweight calibration head with temperature scaling and proper scoring.<n>Experiments on short-form QA, code generation with execution tests, and retrieval-augmented long-form QA show consistent improvements in calibration metrics.
  arXiv  Detail & Related papers  (2025-09-01T13:14:58Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification [118.03466985807331]
  We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence.<n>We introduce the feedback-based evidence retriever(FER) that optimize the evidence retrieval process by incorporating feedback from the claim verifier.
  arXiv  Detail & Related papers  (2023-10-18T02:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.