Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs

• URL: http://arxiv.org/abs/2607.13080v1
• Date: Mon, 13 Jul 2026 07:04:49 GMT
• Title: Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
• Abstract summary: We study a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo.<n>We find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective $0.57 per million tokens.
• Score: 0.5352699766206807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous coding agents force engineering organizations to choose between API-based frontier models -- strong reasoning at high token cost -- and on-premise quantized open-weights models, which promise low-marginal-cost scaling and data sovereignty at some loss of reasoning fidelity. We study this trade-off through a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo: an API-based Claude Opus 4.7/4.8 configuration using Claude Code versus an on-premise GLM-5.1/5.2 configuration using Opencode, quantized to NVFP4, on NVIDIA Blackwell hardware. Analyzing LLM telemetry and Git history, we find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective \$0.57 per million tokens -- below even the \$2.83 amortized unit cost of the shared on-premise slice (a utilization-dependent inversion; total realized spend and total cost of ownership (TCO) are the robust quantities). At comparable gross code churn, the local configuration was associated with a far higher defect-repair burden: a Fix Commit Ratio (FCR) of 74.9% versus 45.9%, with the odds of a commit being a repair 2.6 to 4.9 times higher within every difficulty tier (Mantel-Haenszel OR = 3.61). Under Taiwan-market parameters and a symmetric labor model, on-premise deployment nonetheless saves 40.1% of true TCO under shared GPU allocation, whereas dedicated reservation costs 43.8% more than the cached API. Under shared allocation, the genuine penalty is not monetary but a measurable developer-experience burden -- timestamp indicators show more work trapped in debugging spirals and a slower commit cadence -- and an offline replay shows hybrid routing gateways trade defect rate for infrastructure savings along a cost-quality frontier rather than dominate the pure-API baseline.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• CogEvol: Towards Efficient and Reliable Learning Environment Generation [49.18054685846454]
  We present CogEvol, a family of models trained specifically for Learning Environment Generation.<n>CogEvol completes a slide in a median of 17 seconds and an interactive page in 59, replacing minutes-long multi-turn agent scaffolding.<n>A production-grounded data pipeline turns real failures into 53,687 verified SFT samples.
  arXiv  Detail & Related papers  (2026-08-31T15:33:38Z)
• Data Turnstile: A Scalable Open Framework for Function-Calling Data Generation [4.212647848163336]
  We present Data Turnstile, an open-source framework that generates high-quality synthetic training data for function calling.<n>Turnstile decomposes multi-turn tool-use interactions into constrained, stepwise generation with validation and error-feedback loops.<n>We demonstrate effectiveness of domain adaptation with Turnstile data on two challenging function calling benchmarks.
  arXiv  Detail & Related papers  (2026-07-31T10:21:36Z)
• Token Reduction Is Not Cost Reduction [0.0]
  We show that prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost.<n>Local payload reduction was not a reliable predictor of end-to-end billed cost.<n>We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.
  arXiv  Detail & Related papers  (2026-07-13T21:10:22Z)
• LocalNav: Distilling Frontier VLMs and Embodied RL for On-Device Object Goal Navigation [53.88286427894783]
  Vision Language Models (VLMs) have emerged in the robotic domain as a powerful tool that enables environmental perception with language context.<n>Yet, their computational footprint typically confines them to cloud execution, hindering low-latency inference with local deployment on resource-constrained robots.<n>We present a distillation strategy that transfers complex spatial-semantic reasoning from large frontier models into a lightweight, 4B- parameter local VLM for edge execution on embedded GPU devices.
  arXiv  Detail & Related papers  (2026-06-26T09:11:59Z)
• VisualClaw: A Real-Time, Personalized Agent for the Physical World [76.89211120690028]
  We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
  arXiv  Detail & Related papers  (2026-06-15T06:58:22Z)
• YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition [78.52944812151664]
  Large language models (LLMs) drive significant financial innovations, yet their deployment is bottlenecked by KV cache memory overhead.<n>We propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline built on the Huawei Ascend ecosystem.
  arXiv  Detail & Related papers  (2026-06-04T08:44:37Z)
• Human-Inspired Memory Architecture for LLM Agents [0.9507070656654629]
  We present a biologically-grounded memory architecture comprising six cognitive mechanisms.<n>Each mechanism addresses a specific failure mode of naive memory accumulation.<n>At S-tier scale (50 sessions), dedup-based consolidation yields a +13.3 pp improvement in preference recall.
  arXiv  Detail & Related papers  (2026-05-08T22:52:37Z)
• RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization [13.891522069967507]
  RouteNLP is a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints.<n>In an 8-week pilot deployment processing 5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms.
  arXiv  Detail & Related papers  (2026-04-26T07:34:36Z)
• Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving [9.457255218406333]
  Production vLLM fleets typically provision each instance for the worst-case context length.<n>In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts.<n>We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools.
  arXiv  Detail & Related papers  (2026-04-09T10:47:20Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• The Price Reversal Phenomenon: When Cheaper Reasoning Models End Up Costing More [76.93600828673503]
  Listed API pricing is an unreliable proxy for actual cost.<n>Removing thinking token costs reduces ranking reversals by 70%.<n>Our findings call for cost-aware model selection and transparent per-request cost monitoring.
  arXiv  Detail & Related papers  (2026-03-25T06:07:39Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.