SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

• URL: http://arxiv.org/abs/2609.00035v1
• Date: Sat, 29 Aug 2026 07:07:28 GMT
• Title: SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools
• Abstract summary: We find that machine-checkable constraints yielded an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61.<n>A vocabulary that the description merely exemplifies was missed by every model on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An LLM agent calling a production API cannot distinguish a query that matched nothing from a query the server did not understand. Both return HTTP 200 with a parsable body, no exception to catch and no field to branch on. We ask what predicts which one occurred, and what it does to the agent. Auditing 721,320 parameters across 2,501 independently published OpenAPI documents, we find that 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose that their schema does not encode. Executing 219 schema-derived perturbations against live commercial endpoints from 27 vendors, reached through a single aggregation layer (Monid) that publishes a schema and returns a run identifier for every call, we find that constraint form, not vendor identity, predicts honesty: machine-checkable constraints yielded an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints on ordinary tasks. A vocabulary that the description merely exemplifies was missed by every model on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the resulting silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89. The fix is one line of schema rather than a better model. Code, schemas, perturbation sets, agent transcripts and per-call run identifiers are released at https://github.com/Jasper0122/silentprobe.

Related papers

• Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return [7.146717861821737]
  Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one.<n>We isolate this post-failure decision with a benchmark of 1,024 items spanning 16 internal-system domains and eight tool-failure types.
  arXiv  Detail & Related papers  (2026-09-13T19:46:42Z)
• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows [23.621239974604205]
  APIFlow-Bench is a fully auditable benchmark for long-horizon dependent REST-API performance.<n>We generate synthetic API worlds forward, subtask by subtask.<n>We release all answer keys and 44,362 unredacted execution transcripts.
  arXiv  Detail & Related papers  (2026-08-29T08:12:11Z)
• Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets [13.85834524293015]
  We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with.<n>Models judge whether a candidate belongs far better than they author the set itself.
  arXiv  Detail & Related papers  (2026-08-02T05:00:44Z)
• Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents [48.967927222079965]
  Ledger is a runtime layer that distills an agent's completed interactions into an explicit execution state.<n> Ledger keeps this state in an online execution ledger and applies it at two boundaries of every step.<n>Across all 500 SWE-bench Verified instances, Ledger raises Pass@1 from 56.2% to 64.2% with GPT-5 mini and from 75.8% to 81.0% with MiniMax M2.5.
  arXiv  Detail & Related papers  (2026-08-01T18:25:00Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents [1.6567880228735357]
  In policy-permissive environments, a tool may execute any well-formed call even when the corresponding state transition is forbidden by domain policy.<n>We study this failure mode in the $2$-bench airline domain.<n>We evaluate a lightweight intervention: deterministic, read-only pre-execution gates.
  arXiv  Detail & Related papers  (2026-07-08T13:38:54Z)
• CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment [0.0]
  We place a pre-execution gate between query generation and a production Neo4j database.<n>The gate validates structure through a four-backend chain culminating in execution against a mirror graph at 5.6 ms median latency.<n>On seven CypherBench schemas (2348 questions, ACL 2025) the pipeline maintains generation accuracy on every model tested, confirming it operates as a safe defensive layer.
  arXiv  Detail & Related papers  (2026-06-03T09:13:05Z)
• Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks [40.270213696031625]
  OverEager-Gen is a benchmark dedicated to overeager behavior on benign tasks.<n>On Claude Code, stripping the consent declaration alone raises the overeager rate from 0.0% to 17.1%.<n>OverEager-Gen therefore certifies each scenario's discriminative power before admission.
  arXiv  Detail & Related papers  (2026-05-18T16:00:41Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.