Beyond Benchmarks: The Economics of AI Inference

• URL: http://arxiv.org/abs/2510.26136v1
• Date: Thu, 30 Oct 2025 04:49:27 GMT
• Title: Beyond Benchmarks: The Economics of AI Inference
• Abstract summary: The inference cost of Large Language Models (LLMs) has become a critical factor in determining their commercial viability and widespread adoption.<n>We analyze its marginal cost, economies of scale, and quality of output under various performance configurations.
• Score: 10.810270375582137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The inference cost of Large Language Models (LLMs) has become a critical factor in determining their commercial viability and widespread adoption. This paper introduces a quantitative ``economics of inference'' framework, treating the LLM inference process as a compute-driven intelligent production activity. We analyze its marginal cost, economies of scale, and quality of output under various performance configurations. Based on empirical data from WiNEval-3.0, we construct the first ``LLM Inference Production Frontier,'' revealing three principles: diminishing marginal cost, diminishing returns to scale, and an optimal cost-effectiveness zone. This paper not only provides an economic basis for model deployment decisions but also lays an empirical foundation for the future market-based pricing and optimization of AI inference resources.

Related papers

• The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs [12.942771885641719]
  Inference-time scaling has emerged as a critical avenue for enhancing Large Language Models' performance.<n>In this work, we formulate inference budget allocation as a global constrained optimization problem governed by economic principles.
  arXiv  Detail & Related papers  (2026-06-02T03:26:55Z)
• A Theory of Training Profit-Optimal LLMs [12.802715423118316]
  We develop an economic model for characterizing the rational behavior of an LLM training firm.<n>We analyze the profit problem for this model under compute-bound and data-bound regimes.
  arXiv  Detail & Related papers  (2026-05-14T18:57:40Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Budget-Aware Anytime Reasoning with LLM-Synthesized Preference Data [57.996437077411315]
  We study the reasoning behavior of large language models (LLMs) under limited computation budgets.<n>We introduce an anytime reasoning framework and the Anytime Index, a metric that quantifies how effectively solution quality improves as reasoning tokens increase.<n> Experiments on NaturalPlan (Trip), AIME, and GPQA datasets show consistent gains across Grok-3, GPT-oss, GPT-4.1/4o, and LLaMA models.
  arXiv  Detail & Related papers  (2026-01-16T07:09:30Z)
• A Sustainable AI Economy Needs Data Deals That Work for Generators [56.949279542190084]
  We argue that the machine learning value chain is structurally unsustainable due to an economic data processing inequality.<n>We analyze 73 public data deals and show that the majority of value accrues to aggregators.<n>We propose an Equitable Data-Value Exchange Framework to enable a minimal market that benefits all participants.
  arXiv  Detail & Related papers  (2026-01-15T01:05:48Z)
• The Economics of Information Pollution in the Age of AI: A General Equilibrium Approach to Welfare, Measurement, and Policy [4.887749221165767]
  The advent of Large Language Models (LLMs) represents a fundamental shock to the economics of information production.<n>By asymmetrically collapsing the marginal cost of generating low-quality, synthetic content while leaving high-quality production costly, AI systematically incentivizes information pollution.<n>This paper develops a general equilibrium framework to analyze this challenge.
  arXiv  Detail & Related papers  (2025-09-17T06:31:17Z)
• Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints [1.00707850217229]
  Large language models (LLMs) are limited by substantial computational cost.<n>We introduce a "computational economics" framework that treats an LLM as an internal economy of resource-constrained agents.<n>We show that when computation is scarce, standard LLMs reallocate attention toward high-value tokens while preserving accuracy.
  arXiv  Detail & Related papers  (2025-08-14T07:55:45Z)
• Economic Evaluation of LLMs [0.9208007322096532]
  We show that reasoning models offer better accuracy-cost tradeoffs as soon as the economic cost of a mistake exceeds $0.01.<n>We find that single large LLMs often outperform cascades when the cost of making a mistake is as low as $0.1.
  arXiv  Detail & Related papers  (2025-07-04T23:16:02Z)
• Cost-of-Pass: An Economic Framework for Evaluating Language Models [25.152801302217693]
  We introduce "cost-of-pass", the expected monetary cost of generating a correct solution.<n>We then define the "frontier cost-of-pass" as the minimum cost-of-pass achievable across available models or the "human-expert"<n>We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quantitative, knowledge-intensive, and complex quantitative tasks.
  arXiv  Detail & Related papers  (2025-04-17T21:58:29Z)
• Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models [51.85792055455284]
  Recent advancements in Large Language Models (LLMs) have significantly enhanced their ability to perform complex reasoning tasks.<n>System 1 reasoning is computationally efficient but leads to suboptimal performance.<n>System 2 reasoning often incurs substantial computational costs due to its slow thinking nature and inefficient or unnecessary reasoning behaviors.
  arXiv  Detail & Related papers  (2025-03-31T17:58:07Z)
• Reward-Guided Speculative Decoding for Efficient LLM Reasoning [80.55186052123196]
  We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs)<n>RSD incorporates a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness.<n>RSD delivers significant efficiency gains against decoding with the target model only, while achieving significant better accuracy than parallel decoding method on average.
  arXiv  Detail & Related papers  (2025-01-31T17:19:57Z)
• The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility? [54.18519360412294]
  Large Language Models (LLMs) must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility.<n>This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance.<n>We analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model.
  arXiv  Detail & Related papers  (2025-01-20T06:35:01Z)
• VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback [2.07180164747172]
  This paper addresses the cost-efficiency aspect of Reinforcement Learning from Human Feedback (RLHF)<n>RLHF leverages datasets of human preferences over outputs of large language models (LLM)<n>We show that introducing an auction mechanism can play an essential role in enhancing the cost-efficiency of RLHF.
  arXiv  Detail & Related papers  (2024-09-27T03:15:07Z)
• Benchmarking PtO and PnO Methods in the Predictive Combinatorial Optimization Regime [59.27851754647913]
  Predictive optimization is the precise modeling of many real-world applications, including energy cost-aware scheduling and budget allocation on advertising. We develop a modular framework to benchmark 11 existing PtO/PnO methods on 8 problems, including a new industrial dataset for advertising. Our study shows that PnO approaches are better than PtO on 7 out of 8 benchmarks, but there is no silver bullet found for the specific design choices of PnO.
  arXiv  Detail & Related papers  (2023-11-13T13:19:34Z)
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models [48.87381259980254]
  We document the capability of large language models (LLMs) like ChatGPT to predict stock market reactions from news headlines without direct financial training.<n>Using post-knowledge-cutoff headlines, GPT-4 captures initial market responses, achieving approximately 90% portfolio-day hit rates for the non-tradable initial reaction.
  arXiv  Detail & Related papers  (2023-04-15T19:22:37Z)
• Cost-Sensitive Portfolio Selection via Deep Reinforcement Learning [100.73223416589596]
  We propose a cost-sensitive portfolio selection method with deep reinforcement learning. Specifically, a novel two-stream portfolio policy network is devised to extract both price series patterns and asset correlations. A new cost-sensitive reward function is developed to maximize the accumulated return and constrain both costs via reinforcement learning.
  arXiv  Detail & Related papers  (2020-03-06T06:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.