Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference

• URL: http://arxiv.org/abs/2605.04341v1
• Date: Tue, 05 May 2026 22:59:14 GMT
• Title: Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
• Abstract summary: Budgeted LoRA is a framework that treats model compression as a structured compute allocation problem.<n>We show that Budgeted LoRA matches standard LoRA perplexity at a moderate budget with a 1.74x compressed-module speedup.<n>It achieves a 4.05x speedup with moderate perplexity degradation, and it preserves higher accuracy on function-style in-context learning probes.
• Score: 6.886536285117155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study distillation for large language models under explicit compute constraints, with the goal of producing student models that are not only cheaper to train, but structurally efficient at inference time. While prior approaches to parameter-efficient distillation, such as LoRA, reduce adaptation cost, they leave the dense backbone unchanged and therefore fail to deliver meaningful inference savings. We propose Budgeted LoRA, a distillation framework that treats model compression as a structured compute allocation problem. Instead of using a fixed student architecture, we introduce a global compute budget that sets the final target fraction of dense computation retained. Under this constraint, the model redistributes capacity across dense and low-rank pathways via (i) module-level dense retention coefficients, (ii) adaptive low-rank allocation, and (iii) post-training compression that selectively removes, approximates, or preserves dense components. This formulation yields a family of students controlled by a single budget dial. Empirically, Budgeted LoRA matches standard LoRA perplexity at a moderate budget with a 1.74x compressed-module speedup; at an aggressive budget it achieves a 4.05x speedup with moderate perplexity degradation, and it preserves higher accuracy on function-style in-context learning probes. These results suggest that, under compute-constrained distillation, retaining behavior is less about matching perplexity or removing more parameters than it is about controlling how dense computation is transferred to low-rank pathways.

Related papers

• ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation [4.391214332238358]
  Decision-support in digital agriculture requires accurate predictions and well-calibrated uncertainty estimates.<n>We propose ST-LoRA, a parameter-efficient ensemble framework that builds diverse ensemble members from a single training trajectory.<n> ST-LoRA matches or exceeds full-rank ensembles in segmentation accuracy and calibration across both datasets and architectures.
  arXiv  Detail & Related papers  (2026-08-02T22:49:28Z)
• End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference [40.35003253838957]
  Large Language Models (LLMs) inference is typically deployed under a static resource assumption.<n>We propose Learning to Allocate (L2A), an end-to-end framework for resource-adaptive inference.
  arXiv  Detail & Related papers  (2026-06-26T05:48:21Z)
• Budget Constraints as Riemannian Manifolds [39.54576236079211]
  A recurring problem in machine learning is assigning one of K options to each of N groups under a total cost budget.<n>We propose a new type of constraint that enables first-order optimization of the true objective under exact budget enforcement.<n>On synthetic knapsack problems with known optima, the manifold-based constraint handling recovers optimal solutions, whereas penalty methods plateau at 83% of optimal.
  arXiv  Detail & Related papers  (2026-05-01T13:30:23Z)
• Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization [18.737087162461563]
  Test-time compute scaling has become a powerful lever for improving large language model performance.<n>Yet deploying these techniques under finite inference budgets requires a decision that current systems largely ignore.<n>We formalize this as a constrained optimization problem (maximize expected accuracy subject to an average compute budget) and solve it with a two-stage Solve-then-Learn pipeline.
  arXiv  Detail & Related papers  (2026-04-16T10:39:22Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• BARD: budget-aware reasoning distillation [25.725960386304646]
  Long Chain-of-Thought (CoT) distillation effectively transfers reasoning capability to smaller language models.<n>We propose bftextBudget-Aware Reasoning Distillation (BARD), a novel framework that simultaneously distills reasoning capability and enables fine-grained control over the reasoning length.
  arXiv  Detail & Related papers  (2025-11-03T11:30:18Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  EoRA is a fine-tuning-free method that augments compressed Large Language Models with low-rank matrices.<n>EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression [4.869958813908807]
  We introduce a novel, automatic, and budget-aware rank selection method for efficient model compression.<n> Layer-Wise Imprinting Quantitation (LWIQ) quantifies each layer's significance within a neural network by integrating a proxy classifier.<n> Experimental results on the CIFAR-10 dataset show that our LWIQ improved by 63.2% in rank search efficiency.
  arXiv  Detail & Related papers  (2024-08-02T18:47:11Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)
• Quantization for decentralized learning under subspace constraints [61.59416703323886]
  We consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints. We propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates. The analysis shows that, under some general conditions on the quantization noise, the strategy is stable both in terms of mean-square error and average bit rate.
  arXiv  Detail & Related papers  (2022-09-16T09:38:38Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.