HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards

• URL: http://arxiv.org/abs/2608.06012v1
• Date: Thu, 06 Aug 2026 13:15:54 GMT
• Title: HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards
• Abstract summary: HERALD is an offline audit that applies exact same-question interventions.<n>$R[L]$ meets the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue.
• Score: 7.036896099993471
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Search-agent rewards mix answer quality, citation grounding, tool cost, and anti-hacking terms; a high score therefore need not imply that cited evidence was retrieved, and added penalties can cancel. We introduce HERALD, an offline audit that applies exact same-question interventions, separates candidate-visible from oracle information, and enumerates detector contracts before policy optimization. On four Qwen3-8B pools from HotpotQA, 2WikiMultiHopQA, and MuSiQue, $R_0$ rejects search deletion and fake IDs, but a label-free citation-laundering attack succeeds. A complete $2^3$ ablation identifies targeted strengthening of $L$---citing a corpus passage absent from the retrieved evidence---as the observed inclusion-minimal repair: $R[L]$ has zero empirical ASR with a 0.50% one-sided cluster upper bound. The gap persists across pool rules, a visible BM25 attacker, and four models; broader hardening remains vulnerable when the attack removes an oracle support-ID penalty. Under strict 5M-token matched training evaluated on 256 paired questions per benchmark, $R[L]$ meets the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue. Equal-suite citation precision and support recall improve by 2.02 and 1.46 points, unsupported citations fall by 1.69, and laundering attackability falls on 2Wiki and MuSiQue. Natural $L$ is not reduced, and the detector appears in only 18 of 58,368 training trajectories. HERALD thus separates robust scoring, sparse learning signal, and policy transfer.

Related papers

• A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAG [4.879573049082894]
  We release a benchmark and failure-mode-aware evaluation framework for grounded QA under coordinated retrieval poisoning.<n>The framework partitions reader outputs into four mutually exclusive categories (emphgold, emphhijack, emphabstention, emphdrift)<n>We introduce emphpolymorphic sybil poisoning, a coordinated attack class in which $S$ lexically diverse passages jointly support an attacker-chosen target.
  arXiv  Detail & Related papers  (2026-07-04T06:56:06Z)
• Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment [0.7249731529275342]
  Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality.<n>We find that KLD is strongly correlated with benchmark score over the full cohort.<n>However, this relationship collapses to non-significance in the near-baseline silent zone.
  arXiv  Detail & Related papers  (2026-06-17T19:59:26Z)
• Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation [51.56484100374058]
  Previous work found no measurable coverage gain from adding a live Gemini backend behind an active filter.<n>We introduce $L_5$-no-regex -- identical to $L_4$-real (Gemini-2.5-flash, token-budget cap, rate limit, output scrub) but with the nine-pattern filter disabled.<n>We evaluate it against adversarial probes across three sub-languages.
  arXiv  Detail & Related papers  (2026-06-12T18:54:24Z)
• Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasing [51.56484100374058]
  Production LLM applications stack several defense families -- refusal-phrase filters, tokenbudget controls, model allowlists, rate limits, tool-registry authentication -- yetBAS benchmarks report a single aggregate coverage number.<n>We add four-LLM-Top-10-aware agents to a 21-agent baseline scanner and target a lattice of four synthetic LLM endpoints.
  arXiv  Detail & Related papers  (2026-06-01T19:39:25Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• MEMSAD: Gradient-Coupled Anomaly Detection for Memory Poisoning in Retrieval-Augmented Agents [0.0]
  We formalize memory poisoning attacks on retrieval-augmented agents as a Stackelberg game with a unified evaluation framework.<n>We show faithful evaluation increases measured attack success by $4times$ (ASR-R: 0.25 to 1.00$)<n>Our primary contribution is MEMSAD, a calibration-based defense grounded in a gradient coupling.
  arXiv  Detail & Related papers  (2026-05-05T08:15:41Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models [56.055015597319674]
  Reinforcement learning with verifiable rewards (RLVR) is effective to improve the reasoning ability of large language models (LLMs)<n>Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs.<n>We propose textitCo-rewarding, a novel self-supervised RL framework that improves training stability by seeking complementary supervision from another views.
  arXiv  Detail & Related papers  (2025-08-01T08:09:14Z)
• WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation [57.11538133231843]
  Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm generates keyphrases as a set and has achieved competitive performance. We propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism.
  arXiv  Detail & Related papers  (2022-11-13T09:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.