API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

• URL: http://arxiv.org/abs/2609.08861v2
• Date: Tue, 15 Sep 2026 04:39:01 GMT
• Title: API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces
• Abstract summary: Key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems.<n>We find systematic API--interface differences in both accuracy and consistency.<n>For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4.
• Score: 34.03423347846195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

Related papers

• What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations) [1.0499611180329804]
  We compare two modalities, ChatGPT's chat UI and OpenAI's API, with and without web search enabled.<n>For instance, chat UI responses were less accurate than API responses on both benchmarks with search disabled.<n>We argue that AI safety evaluations should systematically account for modality, multi-run consistency, search conditions, and response-level behaviors.
  arXiv  Detail & Related papers  (2026-08-06T15:58:32Z)
• FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search [88.16262636915975]
  FineVerify is a fine-grained self-verification framework for agentic search.<n>It decomposes each question into checkable sub-questions, verifies sampled candidates, and selects the candidate with the highest aggregated score.<n>FineVerify consistently outperforms standard scaling baselines.
  arXiv  Detail & Related papers  (2026-05-30T10:21:20Z)
• APISENSOR: Robust Discovery of Web API from Runtime Traffic Logs [10.620853813098503]
  APISENSOR is a black-box API discovery framework that reconstructs application APIs unsupervised.<n>We evaluated APISENSOR across six web applications using over 10,000 runtime requests with simulated mixed-traffic noise.<n>Results demonstrate that APISENSOR significantly improves discovery accuracy, achieving an average Group Accuracy Precision of 95.92% and an F1-score of 94.91%.
  arXiv  Detail & Related papers  (2026-03-25T02:30:45Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces [0.0]
  We run 56 20-turn conversations testing ChatGPT-4o and ChatGPT-5 via both the API and chat interface.<n>We find that ChatGPT-5 displays less sycophancy, escalation, and delusion reinforcement than ChatGPT-4o.<n>Even updated models display substantial levels of negative behaviors, revealing that model improvement does not imply model safety.
  arXiv  Detail & Related papers  (2026-02-20T15:48:56Z)
• What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces? [32.72487312750579]
  We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces.<n>Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting.<n>We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces.
  arXiv  Detail & Related papers  (2026-02-02T04:02:03Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• SEAL: Suite for Evaluating API-use of LLMs [1.2528321519119252]
  SEAL is an end-to-end testbed designed to evaluate large language models in real-world API usage. It standardizes existing benchmarks, integrates an agent system for testing API retrieval and planning, and addresses the instability of real-time APIs.
  arXiv  Detail & Related papers  (2024-09-23T20:16:49Z)
• DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection [55.70982767084996]
  A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. We present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions. DeepfakeBench contains 15 state-of-the-art detection methods, 9CL datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-07-04T01:34:41Z)
• DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing [95.18236298557721]
  We present DAPPER (Domain AdaPtation Performance EstimatoR) that estimates the adaptation performance in a target domain with unlabeled target data. Our evaluation with four real-world sensing datasets compared against six baselines shows that DAPPER outperforms the state-of-the-art baseline by 39.8% in estimation accuracy.
  arXiv  Detail & Related papers  (2021-11-22T08:49:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.