Guarded Query Routing for Large Language Models

• URL: http://arxiv.org/abs/2505.14524v2
• Date: Thu, 22 May 2025 07:29:24 GMT
• Title: Guarded Query Routing for Large Language Models
• Authors: Richard Šléher, William Brach, Tibor Sloboda, Kristián Košťál, Lukas Galke,
• Abstract summary: We first introduce the Guarded Query Routing Benchmark (GQR-Bench), which covers three exemplary target domains (law, finance, and healthcare)<n>We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms.<n>Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (4ms)
• Score: 3.1457219084519004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be questions about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench), which covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (<4ms). The embedding-based fastText excels at speed (<1ms) with acceptable accuracy (80%), whereas LLMs yield the highest accuracy (91%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. GQR-Bench will be released as a Python package -- gqr.

Related papers

• Query Routing for Retrieval-Augmented Language Models [38.05904245087491]
  Retrieval-Augmented Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks.<n>We observe that external documents dynamically affect LLM's ability to answer queries, while existing routing methods exhibit suboptimal performance in RAG scenarios.<n>We propose RAG, a parametric RAG-aware routing design, which leverages document embeddings and RAG capability embeddings with contrastive learning to capture knowledge representation shifts.
  arXiv  Detail & Related papers  (2025-05-29T03:44:56Z)
• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing [70.35888047551643]
  We present LaRA, a novel benchmark specifically designed to rigorously compare RAG and LC LLMs.<n>LaRA encompasses 2326 test cases across four practical QA task categories and three types of naturally occurring long texts.<n>We find that the optimal choice between RAG and LC depends on a complex interplay of factors, including the model's parameter size, long-text capabilities, context length, task type, and the characteristics of the retrieved chunks.
  arXiv  Detail & Related papers  (2025-02-14T08:04:22Z)
• Universal Model Routing for Efficient LLM Inference [69.86195589350264]
  Model routing is a technique for reducing the inference cost of large language models (LLMs)<n>We propose UniRoute, a new approach to the problem of dynamic routing, where new, previously unobserved LLMs are available at test time.<n>We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• Reasoning Robustness of LLMs to Adversarial Typographical Errors [49.99118660264703]
  Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting. We study the reasoning robustness of LLMs to typographical errors, which can naturally occur in users' queries. We design an Adversarial Typo Attack ($texttATA$) algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking.
  arXiv  Detail & Related papers  (2024-11-08T05:54:05Z)
• Traceable LLM-based validation of statements in knowledge graphs [0.0]
  This article presents a method for verifying RDF triples using LLMs.<n>Because the LLMs cannot currently reliably identify the origin of the information used to construct the response to the user prompt, our approach is to avoid using internal LLM factual knowledge altogether.<n>Instead, verified RDF statements are compared to chunks of external documents retrieved through a web search or Wikipedia.
  arXiv  Detail & Related papers  (2024-09-11T12:27:41Z)
• QuickLLaMA: Query-aware Inference Acceleration for Large Language Models [94.82978039567236]
  We introduce Query-aware Inference for Large Language Models (Q-LLM) Q-LLM is designed to process extensive sequences akin to human cognition. It can accurately capture pertinent information within a fixed window size and provide precise answers to queries.
  arXiv  Detail & Related papers  (2024-06-11T17:55:03Z)
• Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments [40.95811668230818]
  We propose Reasoning-Path-Editing (Readi) to efficiently reason over structured environments. Readi generates a reasoning path given a query, and edit the path only when necessary. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi.
  arXiv  Detail & Related papers  (2024-03-13T14:59:07Z)
• Autonomous Tree-search Ability of Large Language Models [58.68735916408101]
  Large Language Models have excelled in remarkable reasoning capabilities with advanced prompting techniques. Recent works propose to utilize external programs to define search logic, such that LLMs can perform passive tree search to solve more challenging reasoning tasks. We propose a new concept called autonomous tree-search ability of LLM, which can automatically generate a response containing search trajectories for the correct answer.
  arXiv  Detail & Related papers  (2023-10-14T14:14:38Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.