BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference

• URL: http://arxiv.org/abs/2606.09514v1
• Date: Mon, 08 Jun 2026 14:06:35 GMT
• Title: BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference
• Abstract summary: Buddy is a budget-driven dynamic depth routing framework for large language models.<n>It supports strict budget control, decode-time rerouting, and multiple budgets within a single trained model.
• Score: 8.041559188948003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) incur high inference cost due to their depth and parameter scale. Depth pruning can reduce latency by skipping redundant Transformer blocks, but existing methods (i) provide limited control under user-specific compute budgets and (ii) typically fix the routing path, failing to adapt as the context grows during decoding. We propose Buddy, a budget-driven dynamic depth routing framework. Buddy uses a lightweight Decision Module to score intermediate layers conditioned on the input and deterministically executes the top-k layers to satisfy a given budget. To support decode-time adaptation, Buddy reuses the first-layer KV cache as a low-overhead global context source and pools it together with the newest token representation before each routing decision. When no explicit budget is provided, an optional Budget Predictor estimates an input-dependent compute level to balance quality and efficiency. Experiments on Llama-family and Qwen models show that Buddy is competitive with strong static pruning baselines and often improves the accuracy-compute trade-off, while uniquely supporting strict budget control, decode-time rerouting, and multiple budgets within a single trained model.

Related papers

• BAG: Budget-Aware Gating for Diffusion Caching [13.854727125627107]
  BAG (Budget-Aware Gating) is a novel caching policy that unifies global budget pacing with dynamic, instance-adaptive feature reuse.<n>We train this policy via offline-to-online schedule distillation, transferring the decision-making of offline-searched schedules into a compact online gate.
  arXiv  Detail & Related papers  (2026-08-10T07:54:28Z)
• Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing [5.00725149774706]
  Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer wait time are included.<n>We present Task-to-Model Optimization (T2MO), a data-driven methodology for optimizing model selection in production coding.<n>We treat each developer session as a task that can be discovered, classified, graded for difficulty, benchmarked in a production-like harness, and routed to the cheapest model able to complete it within quality and latency constraints.
  arXiv  Detail & Related papers  (2026-08-09T06:57:53Z)
• End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference [40.35003253838957]
  Large Language Models (LLMs) inference is typically deployed under a static resource assumption.<n>We propose Learning to Allocate (L2A), an end-to-end framework for resource-adaptive inference.
  arXiv  Detail & Related papers  (2026-06-26T05:48:21Z)
• Selective Ensemble Based on Preference-Directed Multi-Objective Bandits [90.75513823660775]
  We formalize a sequential decision problem under partially specified linear preferences.<n>We then propose a preference-directed upper confidence bound (PrefUCB) algorithm.<n> Experiments on large pre-trained model selective ensemble tasks and online asset allocation under institutional mandates validate our method.
  arXiv  Detail & Related papers  (2026-06-20T07:52:46Z)
• Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference [6.886536285117155]
  Budgeted LoRA is a framework that treats model compression as a structured compute allocation problem.<n>We show that Budgeted LoRA matches standard LoRA perplexity at a moderate budget with a 1.74x compressed-module speedup.<n>It achieves a 4.05x speedup with moderate perplexity degradation, and it preserves higher accuracy on function-style in-context learning probes.
  arXiv  Detail & Related papers  (2026-05-05T22:59:14Z)
• Model Compression with Exact Budget Constraints via Riemannian Manifolds [39.54576236079211]
  Assigning one of K options to each of N groups under a total cost budget is a recurring problem in efficient AI.<n>We present a new approach by showing that, under softmax relaxation, the budget constraint defines a smooth Riemannian manifold in logit space with unusually simple geometry.<n>Building on these properties, we propose Riemannian Constrained Optimization (RCO), which augments a standard Adam step with tangent projection, binary-search retraction, and momentum transport.
  arXiv  Detail & Related papers  (2026-05-01T13:30:23Z)
• Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents [40.18150559561834]
  Budget-Aware Value Tree (BAVT) is a training-free inference-time framework that models multi-hop reasoning as a dynamic search tree.<n>BAVT consistently outperforms parallel sampling baselines.
  arXiv  Detail & Related papers  (2026-03-13T04:10:27Z)
• Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory [56.0946692457838]
  BudgetMem is a runtime agent memory framework for explicit, query-aware performance-cost control.<n>A lightweight router performs budget-tier routing across modules to balance task performance and memory construction cost.<n>Across LoCoMo, LongMemEval, and HotpotQA, BudgetMem surpasses strong baselines when performance is prioritized.
  arXiv  Detail & Related papers  (2026-02-05T18:57:09Z)
• e1: Learning Adaptive Control of Reasoning Effort [88.51897900019485]
  Increasing the thinking budget of AI models can significantly improve accuracy, but not all questions warrant the same amount of reasoning.<n>Users may prefer to allocate different amounts of reasoning effort depending on how they value output quality versus latency and cost.<n>We propose Adaptive Effort Control, a self-adaptive reinforcement learning method that trains models to use a user-specified fraction of tokens.
  arXiv  Detail & Related papers  (2025-10-30T23:12:21Z)
• Cost-Aware Contrastive Routing for LLMs [57.30288453580456]
  We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space.<n>CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%.
  arXiv  Detail & Related papers  (2025-08-17T20:16:44Z)
• Scalable Chain of Thoughts via Elastic Reasoning [61.75753924952059]
  Elastic Reasoning is a novel framework for scalable chain of thoughts.<n>It separates reasoning into two phases--thinking and solution--with independently allocated budgets.<n>Our approach produces more concise and efficient reasoning even in unconstrained settings.
  arXiv  Detail & Related papers  (2025-05-08T15:01:06Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.