Routoo: Learning to Route to Large Language Models Effectively

• URL: http://arxiv.org/abs/2401.13979v3
• Date: Wed, 02 Oct 2024 08:51:45 GMT
• Title: Routoo: Learning to Route to Large Language Models Effectively
• Authors: Alireza Mohammadshahi, Arshad Rafiq Shaikh, Majid Yazdani,
• Abstract summary: Routoo is an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routtoo comprises two key components: a performance predictor and cost-aware selector. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third.
• Score: 6.322844087292882
• License:
• Abstract: LLMs with superior response quality--particularly larger or closed-source models--often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement. Routoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality. We evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral's accuracy by over 5% at equivalent costs, achieving an accuracy of 75.9%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4's performance at half the cost and exceeds it with a 25% cost reduction. These outcomes highlight Routoo's potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs.

Related papers

• RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization [53.63439735067081]
  Large language models (LLMs) have achieved impressive performance but face high computational costs and latency. Retrieval-augmented generation (RAG) helps by integrating external knowledge, but imperfect retrieval can introduce distracting noise that misleads SLMs. We propose RoseRAG, a robust RAG framework for SLMs via Margin-aware Preference Optimization.
  arXiv  Detail & Related papers  (2025-02-16T04:56:53Z)
• Cost-Saving LLM Cascades with Early Abstention [1.3108652488669732]
  We investigate the benefits of "early abstention" in LLM cascades. We find that it reduces the overall test loss by 2.2% on average across six benchmarks.
  arXiv  Detail & Related papers  (2025-02-13T08:08:39Z)
• Universal Model Routing for Efficient LLM Inference [72.65083061619752]
  We consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. We prove that these strategies are estimates of a theoretically optimal routing rule, and provide an excess risk bound to quantify their errors.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing [56.98081258047281]
  CITER enables efficient collaboration between small and large language models (SLMs & LLMs) through a token-level routing strategy. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. Our experiments show that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications.
  arXiv  Detail & Related papers  (2025-02-04T03:36:44Z)
• Reward-Guided Speculative Decoding for Efficient LLM Reasoning [80.55186052123196]
  We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs) RSD incorporates a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD delivers significant efficiency gains against decoding with the target model only, while achieving significant better accuracy than parallel decoding method on average.
  arXiv  Detail & Related papers  (2025-01-31T17:19:57Z)
• RouteLLM: Learning to Route LLMs with Preference Data [41.687640419561504]
  Large language models (LLMs) exhibit impressive capabilities across a wide range of tasks, yet the choice of which model to use often involves a trade-off between performance and cost. We propose several efficient router models that dynamically select between a stronger and a weaker LLM during inference. We develop a training framework for these routers leveraging human preference data and data augmentation techniques to enhance performance.
  arXiv  Detail & Related papers  (2024-06-26T18:10:22Z)
• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing [53.748685766139715]
  Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size. We propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.
  arXiv  Detail & Related papers  (2024-04-22T23:06:42Z)
• SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees [21.801053526411415]
  Large Language Models (LLMs) have significantly boosted performance in natural language processing (NLP) tasks. The deployment of high-performance LLMs incurs substantial costs, primarily due to the increased number of parameters aimed at enhancing model performance. We introduce SMART, a novel framework designed to minimize the inference costs of NLP tasks while ensuring sufficient result quality.
  arXiv  Detail & Related papers  (2024-03-11T17:45:47Z)
• Towards Optimizing the Costs of LLM Usage [4.032848774697859]
  We study optimization problems trading off the quality and costs, both theoretically and empirically. We propose several deterministics for reducing tokens in a quality aware manner. Our methods reduce costs by 40%- 90% while improving quality by 4%-7%.
  arXiv  Detail & Related papers  (2024-01-29T16:36:31Z)
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models [53.859446823312126]
  SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks.
  arXiv  Detail & Related papers  (2023-10-11T15:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.