Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models

• URL: http://arxiv.org/abs/2505.20921v2
• Date: Thu, 29 May 2025 05:05:27 GMT
• Title: Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models
• Authors: Injae Na, Keonwoong Noh, Woohwan Jung,
• Abstract summary: We introduce LLM Automatic Transmission (LLM-AT) framework that automatically selects LLM tiers without training.<n>The starter selects the initial LLM tier expected to solve the given question.<n>The generator produces a response using the LLM of the selected tier, and the judge evaluates the validity of the response.
• Score: 1.3791394805787949
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM providers typically offer multiple LLM tiers, varying in performance and price. As NLP tasks become more complex and modularized, selecting the suitable LLM tier for each subtask is a key challenge to balance between cost and performance. To address the problem, we introduce LLM Automatic Transmission (LLM-AT) framework that automatically selects LLM tiers without training. LLM-AT consists of Starter, Generator, and Judge. The starter selects the initial LLM tier expected to solve the given question, the generator produces a response using the LLM of the selected tier, and the judge evaluates the validity of the response. If the response is invalid, LLM-AT iteratively upgrades to a higher-tier model, generates a new response, and re-evaluates until a valid response is obtained. Additionally, we propose accuracy estimator, which enables the suitable initial LLM tier selection without training. Given an input question, accuracy estimator estimates the expected accuracy of each LLM tier by computing the valid response rate across top-k similar queries from past inference records. Experiments demonstrate that LLM-AT achieves superior performance while reducing costs, making it a practical solution for real-world applications.

Related papers

• Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference [7.865726406769634]
  Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy.<n>We introduce LLM Shepherding, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM.<n>Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making.
  arXiv  Detail & Related papers  (2026-01-29T18:52:54Z)
• Neural Bandit Based Optimal LLM Selection for a Pipeline of Tasks [11.389019661082415]
  We propose a neural contextual bandit-based algorithm that trains neural networks that model LLM success on each subtask in an online manner.<n> Experiments on telecommunications question answering and medical diagnosis prediction datasets illustrate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2025-08-13T17:19:41Z)
• LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests [7.6818904666624395]
  This paper proposes a dual-LLM system and experiments with the usage of LLMs for the generation of compiler tests.<n>It is evident that LLMs possess the promising potential to generate quality compiler tests and verify them automatically.
  arXiv  Detail & Related papers  (2025-07-29T02:34:28Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• MixLLM: Dynamic Routing in Mixed Large Language Models [57.309520357563215]
  Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency.<n>We develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment.
  arXiv  Detail & Related papers  (2025-02-09T02:26:15Z)
• Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference [63.03859517284341]
  An automatic evaluation framework aims to rank LLMs based on their alignment with human preferences.<n>An automatic LLM bencher consists of four components: the input set, the evaluation model, the evaluation type and the aggregation method.
  arXiv  Detail & Related papers  (2024-12-31T17:46:51Z)
• PickLLM: Context-Aware RL-Assisted Large Language Model Routing [0.5325390073522079]
  PickLLM is a lightweight framework that relies on Reinforcement Learning (RL) to route on-the-fly queries to available models.<n>We demonstrate the speed of convergence for different learning rates and improvement in hard metrics such as cost per querying session and overall response latency.
  arXiv  Detail & Related papers  (2024-12-12T06:27:12Z)
• Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks [9.867695275243879]
  Large language models (LLMs) have quickly emerged as practical and versatile tools.<n>We consider the application of LLMs on symmetric tasks where a query is asked on an (unordered) bag of elements.
  arXiv  Detail & Related papers  (2024-11-30T17:39:59Z)
• Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval [55.63711219190506]
  Large language models (LLMs) often struggle with posing the right search queries. We introduce $underlineLe$arning to $underlineRe$trieve by $underlineT$rying (LeReT) LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%.
  arXiv  Detail & Related papers  (2024-10-30T17:02:54Z)
• MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs [20.793892860721712]
  We introduce MetaLLM, a framework that dynamically and intelligently routes each query to the optimal large language models (LLMs)<n>By framing the selection problem as a multi-armed bandit, MetaLLM balances prediction accuracy and cost efficiency under uncertainty.<n>Our experiments, conducted on popular LLM platforms such as OpenAI and Together AI, showcase MetaLLM's efficacy in real-world scenarios.
  arXiv  Detail & Related papers  (2024-07-15T15:45:07Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.