What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

• URL: http://arxiv.org/abs/2609.06147v1
• Date: Sat, 05 Sep 2026 15:43:05 GMT
• Title: What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark
• Abstract summary: We built a benchmark of 60 tasks and 470 items from real venture-financing filings to measure how often this happens.<n>We audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown.<n>We release the corpus, all 112,800 raw responses, and the audit as a runnable check for any document benchmark.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown. The audit flags 36 items and separates two failures a single flag would conflate: evidence genuinely absent from the window and answers that must be computed from numbers the window does supply. Flagged items change their answers far more often, wobbling at 0.255 against 0.087 on the 427 clean items, and excluding them cuts apparent cross-model agreement by about a fifth. Before testing whether the missing evidence explains the instability, we registered a prediction: re-cut each window to hold its evidence, and instability should fall below a set threshold. It failed: the repair moved wobble by 0.058, with an interval containing zero. We report the association as correlational. Almost all measurements sit where instability cannot show, which bounds what a corpus built for accuracy can say about stability. We release the corpus, all 112,800 raw responses, and the audit as a runnable check for any document benchmark.

Related papers

• You've Got a BUD in Me: Authenticated Reads from Per-Block Write Logs [41.99844472131922]
  We show how validators can support historical membership and exclusion proofs by authenticating each block's writes instead.<n>A Block Update Digest commits a write log whose predecessor pointers link successive modifications of each key.<n>A SuperBUD summarizes last writes over a window; an exponential hierarchy turns long unchanged intervals into short proofs.
  arXiv  Detail & Related papers  (2026-09-10T08:46:09Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction [6.419341495897254]
  We log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all.<n>We build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value.
  arXiv  Detail & Related papers  (2026-08-28T15:25:12Z)
• TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis [42.62413276719373]
  A trustworthy result should be supported by a valid path from the user question to the relevant data evidence.<n>This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms.<n>We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness.
  arXiv  Detail & Related papers  (2026-08-25T07:09:01Z)
• From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering [0.0]
  We measure, from public material only, whether AI records can express a claim and whether they hold where declared.<n>Across 188 double-graded cells we found no platform whose default record emits a content identity of a behavioral system.<n>An instrument computes realized assurance depth from a pipeline's published exhaust alone and compares it with the declared depth.
  arXiv  Detail & Related papers  (2026-08-21T16:45:00Z)
• Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search [40.897106650424035]
  The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus.<n>We introduce $textBrowseComp-Plus_textCM$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix.<n>Our main contribution is the projection pipeline that makes this possible.
  arXiv  Detail & Related papers  (2026-08-20T17:54:48Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact [0.0]
  AI systems rewrite information constantly: conversations become stored memories, documents become answers.<n>We call that fact-washing, an open-source write-time gate that catches it deterministically, with named flags and evidence rather than an judge.<n>Building it answers a practical question: when does a cheap check suffice, and when do you need a model?
  arXiv  Detail & Related papers  (2026-08-04T09:20:54Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving [13.87786274932945]
  Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.<n>We audit five widely used Lean theorem-proving benchmarks and their forks, using corpus-scale static checkers to surface 4,833 findings.<n>We propose a fault taxonomy, a suite of automated checkers and recall-oriented semantic audit prompts, and release standards to guide the creation of formal math datasets.
  arXiv  Detail & Related papers  (2026-06-28T16:46:36Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.