PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

• URL: http://arxiv.org/abs/2607.20327v1
• Date: Wed, 22 Jul 2026 16:14:26 GMT
• Title: PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
• Abstract summary: PyroDash is a cost-aware framework for token-level SLM-LLM collaborative inference.<n>A Collaborate Engine sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff.
• Score: 0.5568796330843819
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $λ=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $λ=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.

Related papers

• The Embedder's Dilemma: LLMs Are Better, but at What Cost? [18.755750863102232]
  We compare ten text-embedding models with 26 embedding models.<n>The two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro) and the best embedding model (77.2) differ by 0.4 points.<n>Our results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval.
  arXiv  Detail & Related papers  (2026-08-13T06:39:45Z)
• Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference [7.865726406769634]
  Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy.<n>We introduce LLM Shepherding, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM.<n>Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making.
  arXiv  Detail & Related papers  (2026-01-29T18:52:54Z)
• Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval [55.63711219190506]
  Large language models (LLMs) often struggle with posing the right search queries. We introduce $underlineLe$arning to $underlineRe$trieve by $underlineT$rying (LeReT) LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%.
  arXiv  Detail & Related papers  (2024-10-30T17:02:54Z)
• Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement [72.97553348776425]
  We make a pioneering effort to broaden the applicability of merging techniques from FT to PT LLMs. We introduce an approach based on WeIght DisENtanglement (WIDEN) to effectively extend the merging scope. We merge Qwen1.5-Chat (an FT LLM with instruction-following skills) with Sailor (a PT LLM with multilingual abilities) across 7B and 14B model scales.
  arXiv  Detail & Related papers  (2024-08-06T10:46:46Z)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models [79.46938238953916]
  Fine-tuning large language models (LLMs) to diverse applications is crucial to meet complex demands. Recent studies suggest decomposing a fine-tuned LLM into a base model and corresponding delta weights, which are then compressed using low-rank or low-bit approaches to reduce costs. In this work, we observe that existing low-rank and low-bit compression methods can significantly harm the model performance for task-specific fine-tuned LLMs.
  arXiv  Detail & Related papers  (2024-06-13T07:57:27Z)
• How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments [83.78240828340681]
  GAMA($gamma$)-Bench is a new framework for evaluating Large Language Models' Gaming Ability in Multi-Agent environments.<n>$gamma$-Bench includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to assess LLMs' performance.<n>Our results indicate GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought.
  arXiv  Detail & Related papers  (2024-03-18T14:04:47Z)
• Can Large Language Models Play Games? A Case Study of A Self-Play Approach [61.15761840203145]
  Large Language Models (LLMs) harness extensive data from the Internet, storing a broad spectrum of prior knowledge. Monte-Carlo Tree Search (MCTS) is a search algorithm that provides reliable decision-making solutions. This work introduces an innovative approach that bolsters LLMs with MCTS self-play to efficiently resolve turn-based zero-sum games.
  arXiv  Detail & Related papers  (2024-03-08T19:16:29Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)
• Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models [13.799197575126442]
  Small and medium-sized enterprises (SMEs) cannot afford the cost of creating large task-specific training datasets. Third-party services that allow them to prompt Large Language Models currently require a payment per call. We propose a framework that allows reducing the calls to LLMs by caching previous responses and using them to train a local inexpensive model.
  arXiv  Detail & Related papers  (2023-10-20T10:05:07Z)
• Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning [19.472937476936636]
  Large language models (LLMs) have exhibited remarkable performance in a variety of tasks, but this strong performance often comes with the high expense of using paid API services. In this paper, we are motivated to study building an LLM cascade to save the cost of using LLMs. Our proposed cascades can achieve performance comparable to using solely the stronger LLM but require only 40% of its cost.
  arXiv  Detail & Related papers  (2023-10-04T18:21:17Z)
• FLM-101B: An Open LLM and How to Train It with $100K Budget [63.244403881531035]
  We show that FLM-101B, trained with our growth strategy under a budget of $100K, reaches 80% of the baselines' performances with only 10% of their floating-point operations.<n>We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI.
  arXiv  Detail & Related papers  (2023-09-07T17:07:36Z)
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance [36.94826820536239]
  We review the cost associated with querying popular large language models (LLMs) We discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs. Experiments show that FrugalGPT can match the performance of the best individual LLM with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.
  arXiv  Detail & Related papers  (2023-05-09T05:11:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.