External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs

• URL: http://arxiv.org/abs/2606.11806v1
• Date: Wed, 10 Jun 2026 08:38:55 GMT
• Title: External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs
• Abstract summary: Injecting external experience can improve task quality, yet it also increases prompt burden, latency, and serving pressure.<n>We compare no-experience baselines, random experience controls, global prompt injection, and retrieval-based selective injection.
• Score: 3.542522302754046
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Production LLM systems accumulate reusable operational experience, but the practical deployment issue is not merely whether such experience can help. It is how different serving strategies trade off quality against online cost under realistic constraints. Injecting external experience can improve task quality, yet it also increases prompt burden, latency, and serving pressure. We study \textit{external experience serving} as a deployment-oriented quality-cost trade-off problem. We evaluate this question in a real production moderation setting, with tool-use and GPQA as supporting contrast tasks that expose different output-cost regimes. We compare no-experience baselines, random experience controls, global prompt injection, and retrieval-based selective injection, and analyze both task quality and serving cost. The results show that, once experience becomes case-dependent, selective retrieval provides a stronger operating point than unconditional global injection. They further show that retrieval quality matters more than simply increasing Top-$K$, and that the same serving policy can exhibit substantially different cost-benefit profiles across short-output and decode-heavy regimes. These findings suggest that external experience is best treated as a selective, cost-aware serving decision rather than as a universal add-on. Overall, in the settings studied here, external experience pays off only when both the serving interface and the task-specific cost structure make its quality gains worth the online cost.

Related papers

• S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement? [66.17176838165994]
  Large language models (LLMs) increasingly interact with external environments and accumulate substantial behavioral experience.<n>It remains unclear whether an agent can actively test its behavior, judge the resulting experience, and use that experience to improve future decisions.<n>We introduce textbfStextsuperscript3Gym, an interactive benchmark for evaluating LLM self-improvement.
  arXiv  Detail & Related papers  (2026-08-31T17:05:41Z)
• From Novice to Expert: Cost-Aware Bandits for Evolving Worker Performance in Crowdsensing [11.284703747922757]
  We study a budget-constrained online recruitment problem in which the platform selects one worker in each round.<n>We develop a cost-aware online learning framework that jointly learns evolving reward trajectories and heterogeneous costs.
  arXiv  Detail & Related papers  (2026-07-15T07:51:14Z)
• Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applications [2.497641182907452]
  This study examines the impact of RAG and FT on two closed datasets specific to the automotive industry.<n>Our findings reveal that while premium models perform best out of the box, open-source models can achieve comparable quality when enhanced with RAG.
  arXiv  Detail & Related papers  (2026-05-10T13:35:16Z)
• Rethinking Experience Utilization in Self-Evolving Language Model Agents [51.10420305280499]
  Self-evolving agents improve by accumulating and reusing experience from past interactions.<n>This paper studies experience utilization as a critical design dimension of self-evolving agents.
  arXiv  Detail & Related papers  (2026-05-08T02:48:49Z)
• SWE Context Bench: A Benchmark for Context Learning in Coding [6.093520696434546]
  We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate experience reuse in programming agents.<n>Built on SWE-Bench Lite, SWE-ContextBench augments 300 base tasks with 99 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests.<n>We show that correctly selected summarized experience improves resolution accuracy and substantially reduces runtime and token cost.
  arXiv  Detail & Related papers  (2026-02-09T06:44:45Z)
• Large Language Model Agents Are Not Always Faithful Self-Evolvers [84.08646612111092]
  Self-evolving large language model (LLM) agents continually improve by accumulating and reusing past experience.<n>We present the first systematic investigation of experience faithfulness, the causal dependence of an agent's decisions on the experience it is given.
  arXiv  Detail & Related papers  (2026-01-30T01:05:15Z)
• EvoRoute: Experience-Driven Self-Routing LLM Agent Systems [100.64399490164959]
  EvoRoute is a self-evolving model routing paradigm that transcends static, pre-defined model assignments.<n> Experiments on challenging agentic benchmarks demonstrate that EvoRoute, when integrated into off-the-shelf agentic systems, not only sustains or enhances system performance but also reduces execution cost by up to $80%$ and latency by over $70%$.
  arXiv  Detail & Related papers  (2026-01-06T04:06:46Z)
• Self-Refinement Strategies for LLM-based Product Attribute Value Extraction [51.45146101802871]
  This paper investigates applying two self-refinement techniques to the product attribute value extraction task.<n>The experiments show that both self-refinement techniques fail to significantly improve the extraction performance while substantially increasing processing costs.<n>For scenarios with development data, fine-tuning yields the highest performance, while the ramp-up costs of fine-tuning are balanced out as the amount of product descriptions increases.
  arXiv  Detail & Related papers  (2025-01-02T12:55:27Z)
• CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing [70.25689961697523]
  We propose a generalizable algorithm that enhances sequential reasoning by cross-task experience sharing and selection. Our work bridges the gap between existing sequential reasoning paradigms and validates the effectiveness of leveraging cross-task experiences.
  arXiv  Detail & Related papers  (2024-10-22T03:59:53Z)
• Iterative Experience Refinement of Software-Developing Agents [81.09737243969758]
  Large language models (LLMs) can leverage past experiences to reduce errors and enhance efficiency. This paper introduces the Iterative Experience Refinement framework, enabling LLM agents to refine experiences iteratively during task execution.
  arXiv  Detail & Related papers  (2024-05-07T11:33:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.