Token Reduction Is Not Cost Reduction

• URL: http://arxiv.org/abs/2607.12161v2
• Date: Wed, 15 Jul 2026 08:42:05 GMT
• Title: Token Reduction Is Not Cost Reduction
• Abstract summary: We show that prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost.<n>Local payload reduction was not a reliable predictor of end-to-end billed cost.<n>We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Context-reduction layers for API-based coding agents, including command-output compressors, retrieval rankers, and API-boundary proxies, are commonly evaluated by how much context or tool output they remove. We ask a different question: which interventions actually reduce end-to-end billed cost while preserving task success? Our primary evidence is a pre-specified, hash-frozen, paired campaign of 2,908 provider-billed Claude Code runs, of which 2,848 were analyzed, covering 103 tasks, seven repositories, and three models. The campaign compared a baseline with two generations of hook-based compression and an API-boundary proxy within a broader measured program of roughly 5,500 billed executions. Three findings emerge. First, prompt-cache traffic dominated cost composition, accounting for about 87% of reconstructed four-component cost (about 80% of the actual bill), with an 8.7% dollar-weighted residual not attributable from retained telemetry. Second, local payload reduction was not a reliable predictor of end-to-end billed cost. An arm that removed 38% of estimated raw tool-output tokens incurred 6.8% higher paired cost (95% CI: +2.8% to +11.3%), while per-task reduction showed only a weak association with cost change (Pearson r = 0.15). Third, aggressive compression can remove action-critical evidence: on SWE-bench-derived Go tasks, compression reduced successful patch application from 27/40 to 15/40 by corrupting verbatim edit anchors. We propose evaluating context-reduction systems by success-adjusted billed cost rather than token reduction alone.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning [15.652256738395018]
  Large Reasoning Models excel on complex tasks through long chain-of-thought (CoT) reasoning.<n> KV-cache compression is a common solution, yet existing reasoning-oriented methods judge compression only by what it removes from the cache.<n>We propose ReCo, a framework in which a lightweight process-reward estimator scores each completed step and drives three components.
  arXiv  Detail & Related papers  (2026-08-05T12:36:28Z)
• Control Under Compression: Reliability Frontiers for Tool-Using Agents [0.0]
  We introduce CompressAgent, an environment-verified benchmark for ACC compression.<n>At 75% retained context, generic rewriting and section-based compression achieve 92.7% and 92.4% success.<n>At retained-context budgets from 25% to 10%, executable protocols become fragile.
  arXiv  Detail & Related papers  (2026-08-02T07:43:44Z)
• Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs [0.5352699766206807]
  We study a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo.<n>We find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective $0.57 per million tokens.
  arXiv  Detail & Related papers  (2026-07-13T07:04:49Z)
• CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression [77.41779716950387]
  We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement.<n>We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items.
  arXiv  Detail & Related papers  (2026-06-23T02:56:48Z)
• VisualClaw: A Real-Time, Personalized Agent for the Physical World [76.89211120690028]
  We present VisualClaw, a self-evolving multimodal agent built around two principles.<n>First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate.<n>Second, skill evolution lets the agent learn from failures, producing skill-bank updates that help future questions.
  arXiv  Detail & Related papers  (2026-06-15T06:58:22Z)
• Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study [0.0]
  We build a Rust crate that operationalizes affine ownership so that cloning, double-spending, or using a budget after delegating it are compile errors rather than runtime hazards an operator must remember to avoid.<n>Across five runtimes, three providers, and a temperature-stratified live-API test, the approach reports zero cap violations and zero false refusals, at operational parity with concurrent work.
  arXiv  Detail & Related papers  (2026-06-02T10:46:57Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models [9.374717320061988]
  Carbon-Taxed Transformers (CTT) is a systematic multi-architectural compression principled pipeline ordering.<n>CTT operationalizes a computational carbon tax that penalizes architectural inefficiencies and rewards deployment-ready compression.<n>Our results show that CTT delivers on inference: (1) up to 49x memory reduction, (2) time reduction up to 8-10x for clone detection, up to 3x for summarization, and 4-7x for generation.
  arXiv  Detail & Related papers  (2026-04-28T17:48:16Z)
• ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation [9.992155237142306]
  We introduce ClawTrace, an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session.<n>CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches.<n>We show that both cost attribution and prune patches independently reduce quality regressions.
  arXiv  Detail & Related papers  (2026-04-26T19:44:10Z)
• The Price Reversal Phenomenon: When Cheaper Reasoning Models End Up Costing More [76.93600828673503]
  Listed API pricing is an unreliable proxy for actual cost.<n>Removing thinking token costs reduces ranking reversals by 70%.<n>Our findings call for cost-aware model selection and transparent per-request cost monitoring.
  arXiv  Detail & Related papers  (2026-03-25T06:07:39Z)
• Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial [0.05586191108738562]
  The economics of prompt compression depend not only on reducing input tokens but on how compression changes output length.<n>We evaluate this in a pre-registered six-arm randomized controlled trial of prompt compression on production multi-agent task-orchestration.
  arXiv  Detail & Related papers  (2026-03-06T21:47:34Z)
• AgentCompress: Task-Aware Compression for Affordable Large Language Model Agents [0.0]
  A single session using a 70-billion- parameter model can cost around $127 in cloud computing fees.<n>We present AgentCompress, a framework that tackles this problem through task-aware dynamic compression.<n>Results show a 68.3% reduction in computational costs while preserving 96.2% of the original success rate.
  arXiv  Detail & Related papers  (2026-01-08T18:13:46Z)
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient [52.74493506816969]
  Large Reasoning Models excel at complex tasks using Chain-of-Thought (CoT) reasoning.<n>Their tendency to overthinking leads to unnecessarily lengthy reasoning chains.<n>We introduce VeriThinker, a novel approach for CoT compression.
  arXiv  Detail & Related papers  (2025-05-23T14:17:56Z)
• Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression [55.323397702682506]
  Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining.<n>We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery.
  arXiv  Detail & Related papers  (2025-04-10T02:19:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.