Guarded Query Routing for Large Language Models

• URL: http://arxiv.org/abs/2505.14524v3
• Date: Sat, 25 Oct 2025 09:38:24 GMT
• Title: Guarded Query Routing for Large Language Models
• Abstract summary: We study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench)<n>Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (4ms)<n>Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications.
• Score: 2.2349474636722317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a guarded query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench, released as Python package gqr), covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88%) and speed (<4ms). The embedding-based fastText excels at speed (<1ms) with acceptable accuracy (80%), whereas LLMs yield the highest accuracy (91%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. Source code is available: https://github.com/williambrach/gqr.

Related papers

• Your Retriever Already Knows: Distribution-Shape QPP for RAG Retrieval Sufficiency [0.0]
  Motivated by a Czech nuclear-regulator deployment, we compare three Query Performance Prediction (QPP) paradigms for retrieval sufficiency in RAG.<n>Our 24 non-lexical features reach a weighted-average AUROC of 0.856 at 2 ms per query, ahead of a classic-QPP literature pool.<n>Adding the LLM judgment as one feature (hybrid) matches S1 on ViDoRe (0.863) but gains a statistically significant edge on SJB.
  arXiv  Detail & Related papers  (2026-09-10T14:52:16Z)
• Influence of Prompt Engineering on Small Language Models for Guarded Query Routing [0.0]
  We investigate whether compact open-weight Small Language Models (SLMs) can jointly handle both tasks under latency constraints.<n>We find that mid-scale SLMs come close to frontier model routing quality at much lower latency.<n>Results show that prompt optimization techniques enable SLMs to handle such cases gracefully, without changing the models' weights.
  arXiv  Detail & Related papers  (2026-07-06T09:21:18Z)
• LogRouter: Adaptive Two-Level LLM Routing for Log Question Answering in Big Data Systems [0.0]
  We present an end-to-end log question-answering system deployed on TUBITAK BILGEM's national big data platform.<n>A two-level cost-aware router dispatches each query along one of four execution paths.<n>The system reduces mean latency by 55% versus a Fixed-32B baseline.
  arXiv  Detail & Related papers  (2026-05-18T08:07:01Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• SelRoute: Query-Type-Aware Routing for Long-Term Conversational Memory Retrieval [0.0]
  We introduce SelRoute, a framework that routes each query to a specialized retrieval pipeline based on its query type.<n>On LongMemEval_M, SelRoute achieves Recall@5 of 0.800 with bge-base-en-v1.5 (109M parameters) and 0.786 with bge-small-en-v1.5 (109M parameters)<n>A zero-ML baseline using FTS5 alone achieves NDCG@5 of 0.692, already exceeding all published baselines on storage quality.
  arXiv  Detail & Related papers  (2026-04-02T18:02:59Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• Query Routing for Retrieval-Augmented Language Models [38.05904245087491]
  Retrieval-Augmented Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks.<n>We observe that external documents dynamically affect LLM's ability to answer queries, while existing routing methods exhibit suboptimal performance in RAG scenarios.<n>We propose RAG, a parametric RAG-aware routing design, which leverages document embeddings and RAG capability embeddings with contrastive learning to capture knowledge representation shifts.
  arXiv  Detail & Related papers  (2025-05-29T03:44:56Z)
• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing [70.35888047551643]
  We present LaRA, a novel benchmark specifically designed to rigorously compare RAG and LC LLMs.<n>LaRA encompasses 2326 test cases across four practical QA task categories and three types of naturally occurring long texts.<n>We find that the optimal choice between RAG and LC depends on a complex interplay of factors, including the model's parameter size, long-text capabilities, context length, task type, and the characteristics of the retrieved chunks.
  arXiv  Detail & Related papers  (2025-02-14T08:04:22Z)
• Universal Model Routing for Efficient LLM Inference [69.86195589350264]
  Model routing is a technique for reducing the inference cost of large language models (LLMs)<n>We propose UniRoute, a new approach to the problem of dynamic routing, where new, previously unobserved LLMs are available at test time.<n>We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• Reasoning Robustness of LLMs to Adversarial Typographical Errors [49.99118660264703]
  Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning using Chain-of-Thought (CoT) prompting. We study the reasoning robustness of LLMs to typographical errors, which can naturally occur in users' queries. We design an Adversarial Typo Attack ($texttATA$) algorithm that iteratively samples typos for words that are important to the query and selects the edit that is most likely to succeed in attacking.
  arXiv  Detail & Related papers  (2024-11-08T05:54:05Z)
• Traceable LLM-based validation of statements in knowledge graphs [0.0]
  This article presents a method for verifying RDF triples using LLMs.<n>Because the LLMs cannot currently reliably identify the origin of the information used to construct the response to the user prompt, our approach is to avoid using internal LLM factual knowledge altogether.<n>Instead, verified RDF statements are compared to chunks of external documents retrieved through a web search or Wikipedia.
  arXiv  Detail & Related papers  (2024-09-11T12:27:41Z)
• QuickLLaMA: Query-aware Inference Acceleration for Large Language Models [94.82978039567236]
  We introduce Query-aware Inference for Large Language Models (Q-LLM) Q-LLM is designed to process extensive sequences akin to human cognition. It can accurately capture pertinent information within a fixed window size and provide precise answers to queries.
  arXiv  Detail & Related papers  (2024-06-11T17:55:03Z)
• Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments [40.95811668230818]
  We propose Reasoning-Path-Editing (Readi) to efficiently reason over structured environments. Readi generates a reasoning path given a query, and edit the path only when necessary. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi.
  arXiv  Detail & Related papers  (2024-03-13T14:59:07Z)
• Autonomous Tree-search Ability of Large Language Models [58.68735916408101]
  Large Language Models have excelled in remarkable reasoning capabilities with advanced prompting techniques. Recent works propose to utilize external programs to define search logic, such that LLMs can perform passive tree search to solve more challenging reasoning tasks. We propose a new concept called autonomous tree-search ability of LLM, which can automatically generate a response containing search trajectories for the correct answer.
  arXiv  Detail & Related papers  (2023-10-14T14:14:38Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.