Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction

• URL: http://arxiv.org/abs/2606.24420v1
• Date: Tue, 23 Jun 2026 10:58:08 GMT
• Title: Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction
• Abstract summary: ExtractConf is a cross-domain, field-agnostic confidence engine.<n>It grounds confidence estimation in two structurally different readings of the same document.<n>It achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean.
• Score: 2.1997815944423516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In high-stakes document processing pipelines, including financial reconciliation, compliance verification, and procurement automation, an LLM extraction that is silently wrong is more dangerous than one that is visibly absent. The central challenge is not extraction accuracy alone but reliable confidence estimation: knowing, field by field, whether an extraction can be trusted for automation or deferred to human review. Token-level log-probabilities, verbalized confidence, and multi-sample self-consistency all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions. We present ExtractConf, a cross-domain, field-agnostic confidence engine that grounds confidence estimation in two structurally different readings of the same document. A field-guided Hunter call extracts each field under schema-slot completion pressure; a document-guided Mapper call scans holistically and surfaces values grounded in document content. This asymmetry yields different failure modes: Hunter hallucinates values for absent fields, while Mapper misses visually non-salient ones. Their disagreement is independently informative. ExtractConf fuses cross-call disagreement, LLM-internal uncertainty, OCR, image quality, and spatial layout into a classifier requiring no domain-specific rules or retraining. On DocILE (55-field invoices, 26% failure rate), it achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean. At 80% coverage, accuracy reaches 99.1%, enabling a practical human-in-the-loop workflow. Zero-shot transfer to CORD receipts achieves 0.858 AUC; lightweight Lasso recalibration reduces ECE by 89% and Brier by 43%, confirming the signals generalise across document domains.

Related papers

• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• Cross-Fitted Residual Utility for Primary-Preserving Cognitive Decision Correction in Automatic Modulation Classification [1.547549252134621]
  We study a post-inference problem through cross-fitted residual utility and a primary-preserving cognitive decision policy.<n>A structured KAN-Fourier classifier supplies the default probability, while neural and non-neural candidates provide observable evidence.<n>On RMLA, RMLB, and HISAR, the complete system improves overall accuracy from 63.632% to 66.332%, 65.161% to 66.168%, and 77.769% to 79.867%.
  arXiv  Detail & Related papers  (2026-08-03T11:07:37Z)
• Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction [13.176724997257802]
  Existing document benchmarks are dominated by clean, high-quality samples.<n>We introduce ConfBench, the first calibration-specific benchmark for key information extraction.<n>We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods.
  arXiv  Detail & Related papers  (2026-08-03T07:03:51Z)
• Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference [22.335515110971475]
  Visual-token pruning is judged by answer quality at a fixed retention budget.<n>We turn this blind spot into an evidence-risk audit that couples answer behavior with geometric token-origin provenance.<n>We show that favorable target-verification points do not imply task-general compression.
  arXiv  Detail & Related papers  (2026-07-29T13:45:31Z)
• Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoring [1.0170129555792935]
  Diagnosing 802.11 packet captures requires expert protocol knowledge, is slow, inconsistent across engineers, and unscalable.<n>LLM-based approaches sound plausible but fabricate protocol events absent from captures, produce uncalibrated confidence scores, and suffer evaluation bias when golden references are co-produced by the model under test.<n>We introduce PROBE, a multi-stage pipeline addressing all three failures.
  arXiv  Detail & Related papers  (2026-06-05T03:39:58Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• When the Ruler is Broken: Parsing-Induced Suppression in LLM-Based Security Log Evaluation [0.0]
  We demonstrate a class of silent, systematic evaluation errors that can cause a fully functional model to appear completely non-functional categories.<n>We propose SOC-Bench v0, a benchmark framework comprising a standardized 13 category threat taxonomy, minimum statistical power requirements, fuzzy field extraction specification, and a public scoring script intended to prevent specific accuracy distortion in future SOC research.
  arXiv  Detail & Related papers  (2026-05-08T06:03:11Z)
• SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio [0.038379177968040606]
  We propose SELFDOUBT, a single-pass uncertainty framework for reasoning language models.<n>Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior.<n> SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API.
  arXiv  Detail & Related papers  (2026-04-07T19:19:29Z)
• Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection [0.0]
  Large language models (LLMs) are increasingly deployed in critical decision-making systems.<n>We introduce a normalized confidence score based on output anchor token probabilities.<n>This enables direct detection of errors and hallucinations with minimal overhead.
  arXiv  Detail & Related papers  (2026-02-18T07:05:12Z)
• Secure and Explainable Fraud Detection in Finance via Hierarchical Multi-source Dataset Distillation [17.90471000973834]
  A trained random forest is converted into transparent, axis-aligned rule regions.<n>Synthetic transactions are generated by uniformly sampling within each region.<n>This produces a compact, auditable surrogate dataset.
  arXiv  Detail & Related papers  (2025-12-26T05:00:35Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.