Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

• URL: http://arxiv.org/abs/2608.14639v1
• Date: Tue, 28 Jul 2026 17:08:50 GMT
• Title: Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays
• Abstract summary: Per-field accept/review with selective risk at most alpha is the trust contract document-extraction systems need.<n>On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes.<n>Released Apache-2.0 with seed-pinned, regression-gated procedures.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents. On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes: document clustering (design effect 1.84-2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and a tie-mass pathology (a degenerate score collapses the threshold grid, 0.030 to 0.001). We organize the fixes as a validity ladder, guarantee form stated per tier. A fit/val split protocol restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, no tolerance band (production variant 0.326) -- an on-average point whose realized risk exceeds alpha in 47.5% of resplits, not a certificate. Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates: field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060 -- the only tier matching documents, honestly near-vacuous today. Support-bin, the pre-specified provenance taxonomy, wins every rigor tier on the sonnet CORD capture (p<1e-4, Bonferroni-corrected) -- a win that does not replicate on the same documents under haiku or qwen -- while on higher-accuracy corpora pooled thresholds win: conditioning helps exactly where pooled cannot certify, subsumed by a learned score elsewhere. A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83; labels err one-sidedly pessimistic). Released Apache-2.0 with seed-pinned, regression-gated procedures.

Related papers

• Guaranteed Adaptive Modality Acquisition: When the Policy Chooses Its Own Calibration Group [0.0]
  A multimodal system may begin inference holding only some of its inputs and may acquire the rest at a cost.<n>We characterize when pattern-conditional guarantees remain valid and give two finite-sample constructions.<n>A counterexample shows that a guarantee proved for a calibration-independent grouping map need not transfer once the policy makes the terminal group calibration-dependent.
  arXiv  Detail & Related papers  (2026-08-16T04:19:31Z)
• Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify [1.0742675209112622]
  We formulate PRM stress testing as a quality-diversity search problem using MAP-Elites.<n>We characterize what such archives certify: finite-cell repair bounds covered-cell tail risk and average residual severity but cannot bound the worst remaining cell from covered fraction alone.<n>A predeclared paired LoRA repair protocol reduces exploit rates from 0.148 to 0.037 to 0.074, the worst attack from 0.333 to 0.177 to 0.212, improves ranking AUROC without degrading best-of-4 accuracy.
  arXiv  Detail & Related papers  (2026-08-08T08:41:06Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation [0.11280931253550518]
  Large language models (LLMs) deployed for structured generation (NER, extraction, QA, and classification) lack formal reliability guarantees.<n>We characterize when conformal risk control (CRC) can certify structured LLM outputs and when it provably cannot.
  arXiv  Detail & Related papers  (2026-06-27T19:25:07Z)
• False Sense of Safety in Selective Signal Classification: Auditing Bound Tightness and Exchangeability for Risk Control [6.3999417080764225]
  Selective prediction with distribution-free risk control promises that, with confidence 1-delta over the calibration draw, the error rate of accepted inputs stays below a user budget alpha.<n>We audit this promise on signal-domain detectors -- machine anomalous-sound detection (ASD) and AI-generated-image forensics.
  arXiv  Detail & Related papers  (2026-06-13T06:48:50Z)
• A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control [0.9668407688201359]
  We give a certificate for bounded, possibly non-monotone losses by treating the selected risk directly as a ratio.<n>Relative to the range-only Hoeffding-ratio construction this sharpens the acceptance-floor dependence from $1/pmin$ to $1/sqrtpmin$.<n>The certificate opens a $+22$ pp certified-acceptance frontier over Hoeffding--CRC.
  arXiv  Detail & Related papers  (2026-06-07T08:55:44Z)
• Hierarchical Certified Semantic Commitment for Byzantine-Resilient LLM-Agent Collaboration [30.310793549183117]
  We introduce Hierarchical Certified Semantic Commitment (H-CSC), a BFT-inspired protocol that converts embedding-derived finality signals over verdict-emitting proposal groups into one of three typed outcomes.<n>H-CSC commits with low angular deviation on BFT-feasible buckets (0.31 to 2.04 degrees) and aborts 100% of beyond-BFT rounds (n3f+1) as intended.
  arXiv  Detail & Related papers  (2026-06-05T14:35:58Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation [43.148402136307716]
  Cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal.<n>We transcribed 50 publicly available medical audio clips with eight ASR systems spanning commercial APIs and open-source engines.<n>Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass.
  arXiv  Detail & Related papers  (2026-03-02T13:02:13Z)
• Conditional Coverage Diagnostics for Conformal Prediction [47.93989136542648]
  We show that conditional coverage estimation can be a classification problem.<n>We call the resulting family of metrics excess risk of the target coverage (ERT)<n>We release an open-source package for ERT as well as previous conditional coverage metrics.
  arXiv  Detail & Related papers  (2025-12-12T18:47:39Z)
• Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers [34.31192184496381]
  Dense retrievers and rerankers are central to retrieval-augmented generation (RAG) pipelines.<n>We evaluate how much factual competence these components inherit or lose from large language models (LLMs) they are based on.<n>For every embedding model, cosine-similarity scores between queries and correct completions are significantly higher than those for incorrect ones.
  arXiv  Detail & Related papers  (2025-08-28T04:13:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.