Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

• URL: http://arxiv.org/abs/2608.16391v1
• Date: Mon, 17 Aug 2026 10:41:18 GMT
• Title: Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
• Abstract summary: We propose mboxtextbfVentor-QTest, a composite black-box audit that requires no probability information from the target API.<n>Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports emphaverage fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic.
• Score: 5.713807329849712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.

Related papers

• Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing [65.59464371673623]
  Local Branch Routing (LBR) is a token-level test-time scaling framework.<n>It expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit.<n>LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search.
  arXiv  Detail & Related papers  (2026-06-24T03:42:44Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora [0.0]
  We present OrgForge, an open-source multi-agent simulation framework that enforces a strict physics-cognition boundary.<n>An actor-local clock enforces causal timestamp correctness across all artifact types.<n>Running a N-day simulation, OrgForge produces interleaved Slack threads, tickets, Confluence pages, Git pull requests, and emails, all traceable to a shared, immutable event log.
  arXiv  Detail & Related papers  (2026-03-16T09:02:24Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• WuppieFuzz: Coverage-Guided, Stateful REST API Fuzzing [0.0]
  WuppieFuzz is an open-source REST API fuzzer built on LibAFL.<n>It supports white-box, grey-box and black-box fuzzing.
  arXiv  Detail & Related papers  (2025-12-17T16:05:39Z)
• Nondeterminism-Aware Optimistic Verification for Floating-Point Neural Networks [11.159101812934503]
  Nondeterministic Aware Optimistic verification protocol for floating-point execution on heterogeneous accelerators.<n>We implement NAO as a PyTorch-compatible runtime and a contract layer currently deployed on Holesky testnet.
  arXiv  Detail & Related papers  (2025-10-15T21:10:39Z)
• You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs [50.54173262572369]
  Large language models (LLMs) are increasingly deployed in specialized domains such as finance, medicine, and agriculture.<n>We study label-free test-time adaptation for language models and present SyTTA, an inference-time framework that adapts models on-the-fly without additional supervision.
  arXiv  Detail & Related papers  (2025-10-11T14:00:39Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Model Equality Testing: Which Model Is This API Serving? [59.005869726179455]
  API providers may quantize, watermark, or finetune the underlying model, changing the output distribution.<n>We formalize detecting such distortions by Model Equality Testing, a two-sample testing problem.<n>A test built on a simple string kernel achieves a median of 77.4% power against a range of distortions.
  arXiv  Detail & Related papers  (2024-10-26T18:34:53Z)
• Sharp Bounds for Sequential Federated Learning on Heterogeneous Data [5.872735527071425]
  In Federated Learning (FL), models are trained in a parallel manner across clients, and in sequential FL (SFL) models are trained in a sequential manner across clients.<n>In contrast to PFL, convergence theory of SFL data is still lacking.<n>We establish sharp convergence guarantees for SFL on heterogeneous data with both upper and lower bounds.
  arXiv  Detail & Related papers  (2024-05-02T09:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.