White Box Evidence Packages for Policy Audit Reports

• URL: http://arxiv.org/abs/2607.21462v1
• Date: Thu, 23 Jul 2026 16:06:42 GMT
• Title: White Box Evidence Packages for Policy Audit Reports
• Abstract summary: We show that internal evidence changes how reports cite and reason about evidence, but more internal citations do not by themselves make a report more valid.<n>This study reframes internal model access as an evidence design problem for audit, rather than as a guarantee of transparency.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI governance moves from benchmark scores toward auditable oversight, a central question is how reviewers can tell whether an LLM-generated audit report is actually supported by evidence. This paper studies that question in passage-anchored policy audits, where a report must interpret a given policy passage and cite evidence for its claims. We introduce a controlled evaluation framework that holds the passage, rubric, and auditor model fixed while changing only the evidence interface supplied to the auditor. Across 60 AGORA policy cases, we generate 600 structured reports under ten evidence conditions, including passage-based evidence, internal model evidence, a hybrid package, and a shuffled control that preserves evidence format while breaking case relevance. Five human reviewers evaluate the primary interfaces for correctness, passage grounding, diagnostic usefulness, and evidence misuse. The results show that internal evidence changes how reports cite and reason about evidence, but more internal citations do not by themselves make a report more valid. A white-box diagnostic explains the failure mode: causal localization is narrow, while reports readily reuse broader readable labels and token directions. The hybrid interface is the most useful on average, while the shuffled control exposes a key governance risk: reports can sound substantively plausible while citing irrelevant internal evidence. This study reframes internal model access as an evidence design problem for audit workflows, rather than as a guarantee of transparency.

Related papers

• From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents [47.678562263387214]
  Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings.<n>Yet a completed trajectory is not automatically evidence: generated artifacts may be unsupported or incomplete, executed rounds may be invalid or confounded, and later modifications may obscure earlier findings.<n>We introduce an evidence-grounded framework that couples bounded verification of consequential artifacts with post-execution claim qualification.
  arXiv  Detail & Related papers  (2026-08-05T13:37:38Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments [51.049999642138]
  We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
  arXiv  Detail & Related papers  (2026-07-19T15:20:40Z)
• Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits [0.30586855806896046]
  We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models.<n>We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence.
  arXiv  Detail & Related papers  (2026-07-01T05:30:07Z)
• RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild [9.55806677152407]
  RW-Post is a post-aligned textbftext--image benchmark for real-world multimodal fact-checking.<n> RW-Post supports controlled evaluation across closed-book, evidence-bounded, and open-web regimes.
  arXiv  Detail & Related papers  (2026-05-11T11:04:04Z)
• Case-Grounded Evidence Verification: A Framework for Constructing Evidence-Sensitive Supervision [1.7044563989708406]
  We introduce case-grounded evidence verification.<n>We train a standard verifier on the resulting support task.<n>Results suggest that a major bottleneck in evidence grounding is not only model capacity, but the lack of supervision.
  arXiv  Detail & Related papers  (2026-04-10T17:55:38Z)
• FactReview: Evidence-Grounded Reviews with Literature Positioning and Execution-Based Claim Verification [57.196748998757954]
  We present FactReview, an evidence-grounded reviewing system that combines claim extraction, literature positioning, and execution-based claim verification.<n>Given a submission, FactReview identifies major claims and reported results, retrieves nearby work to clarify the paper's technical position, and, when code is available, executes the released repository.<n>It then produces a concise review and an evidence report that assigns each major claim one of five labels: Supported, Supported by the paper, Partially supported, In conflict, or Inconclusive.
  arXiv  Detail & Related papers  (2026-04-05T11:45:22Z)
• D-SECURE: Dual-Source Evidence Combination for Unified Reasoning in Misinformation Detection [9.049034101566642]
  Multimodal misinformation increasingly mixes realistic im-age edits with fluent but misleading text, producing persuasive posts that are difficult to verify.<n>We present D-SECURE, a framework that combines internal manipulation detection with external evidence-based reasoning for news-style posts.
  arXiv  Detail & Related papers  (2026-02-16T03:51:49Z)
• From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification [118.03466985807331]
  We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence.<n>We introduce the feedback-based evidence retriever(FER) that optimize the evidence retrieval process by incorporating feedback from the claim verifier.
  arXiv  Detail & Related papers  (2023-10-18T02:59:38Z)
• Give Me More Details: Improving Fact-Checking with Latent Retrieval [58.706972228039604]
  Evidence plays a crucial role in automated fact-checking. Existing fact-checking systems either assume the evidence sentences are given or use the search snippets returned by the search engine. We propose to incorporate full text from source documents as evidence and introduce two enriched datasets.
  arXiv  Detail & Related papers  (2023-05-25T15:01:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.