Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

• URL: http://arxiv.org/abs/2608.28439v1
• Date: Fri, 28 Aug 2026 15:25:12 GMT
• Title: Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction
• Abstract summary: We log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all.<n>We build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value.
• Score: 6.419341495897254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet's claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window.

Related papers

• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• Twin Worlds: Equivariance-Based Abstention for Evidence-Grounded Reasoning [38.63002496590365]
  We propose a framework for improving reliability in knowledge-intensive reasoning through equivariance-based abstention.<n>A key contributing factor is that entity mentions in context activate memorised associations, causing models to generate plausible responses ungrounded in evidence.<n>We construct multiple worlds via typed substitutions of the original input that preserve relational structure while reducing parametric priors.
  arXiv  Detail & Related papers  (2026-08-28T07:33:29Z)
• ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices [48.36933831982682]
  We introduce ADeptS-Bench, a trustworthiness benchmark grounded in the ADEPTS capability framework.<n>The Safety stream provides paired benign/malicious tasks with threats embedded in the visual interface.<n>The Disambiguation stream evaluates whether agents seek clarification when intent is ambiguous.
  arXiv  Detail & Related papers  (2026-08-25T23:21:57Z)
• From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering [0.0]
  We measure, from public material only, whether AI records can express a claim and whether they hold where declared.<n>Across 188 double-graded cells we found no platform whose default record emits a content identity of a behavioral system.<n>An instrument computes realized assurance depth from a pipeline's published exhaust alone and compares it with the declared depth.
  arXiv  Detail & Related papers  (2026-08-21T16:45:00Z)
• The LLM Proposes, the Executive Disposes: A Self-Verifying Agent Instrument that Dissociates Commitment Drift from Binding Drift in Long-Horizon Agents [0.0]
  We present an agent instrument built so that verification is structural rather than post-hoc.<n>Every run invalidates itself when per-organ write-error, render-size, or salted-canary-echo floors are breached.<n>We report a clean, single-variable result on a failure every long-horizon agent suffers.
  arXiv  Detail & Related papers  (2026-08-04T15:10:37Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI [39.17006317116773]
  We argue that "zero hallucination" is not a property a model possesses but a property a system enforces.<n>We present HALO, an assurance architecture which treats hallucination as a containable failure mode rather than an eliminable one.
  arXiv  Detail & Related papers  (2026-07-20T12:34:44Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs [0.564046562677526]
  We show how a Lean 4-based tool-calling architecture can be used to govern empirical claims.<n>Every verified output structurally descends from an attested tool call and a kernel-checked chain of valid inference.<n>A formal sidecar makes the target proposition, source scope, evidence boundary, proof obligation, and abstention condition auditable.<n>Over time, typed sidecars in datasets, APIs, public records, and AI-generated documents can amortize this formalization burden into reusable infrastructure.
  arXiv  Detail & Related papers  (2026-07-14T11:33:44Z)
• Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis [2.7092559671391]
  Agentic LLM systems such as OpenScholar and PaperQA2 read the scientific literature and return cited answers.<n>Neither audits the reliability of that check itself.<n>We show it is not reliable, and that this matters.<n>We present a gold-anchored evaluation protocol and a deployable guard.
  arXiv  Detail & Related papers  (2026-07-10T02:05:17Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• A measurement substrate for agentic Kubernetes operations: Methodology and a case study in retrieval-compounding falsification [0.0]
  Empirical claims about autonomous operations agents are largely unfalsifiable.<n>Code agents have a verification substrate that turns "did it work" into a fast, falsifiable, groundtruth signal.<n>We present agent-breakage, a closed-loop measurement framework that injects faults into a target cluster.
  arXiv  Detail & Related papers  (2026-05-21T21:47:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.