RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization

• URL: http://arxiv.org/abs/2604.23577v1
• Date: Sun, 26 Apr 2026 07:34:36 GMT
• Title: RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
• Abstract summary: RouteNLP is a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints.<n>In an 8-week pilot deployment processing 5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms.
• Score: 13.891522069967507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Serving diverse NLP workloads with large language models is costly: at one enterprise partner, inference costs exceeded $200K/month despite over 70% of queries being routine tasks well within the capability of smaller models. We present RouteNLP, a closed-loop framework that routes queries across a tiered model portfolio to minimize cost while satisfying per-task quality constraints. The framework integrates three components: a difficulty-aware router with shared task-conditioned representations trained on preference data and quality signals; confidence-calibrated cascading that uses conformal prediction for distribution-free threshold initialization; and a distillation-routing co-optimization loop that clusters escalation failures, applies targeted knowledge distillation to cheaper models, and automatically retrains the router, yielding over twice the cost improvement of untargeted distillation. In an 8-week pilot deployment processing ~5K queries/day at an enterprise customer-service division, RouteNLP reduced inference costs by 58% while maintaining 91% response acceptance and reducing p99 latency from 1,847 ms to 387 ms. On a six-task benchmark spanning finance, customer service, and legal domains, the framework achieves 40-85% cost reduction while retaining 96-100% quality on structured tasks and 96-98% on generation tasks, with human evaluation confirming that 74.5% of routed generation outputs match or exceed frontier-model quality.

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• A Deep Reinforcement Learning Algorithm for the Vehicle Routing Problem with Stochastic Demands and Outsourcing [0.0]
  Vehicle routing problem with demands and outsourcing options (VRP-SDO)<n>We learn an offline routing policy that estimates costs almost instantly for any committed subset.<n> Experiments show that our policy reduces routing costs by 19.6% relative to a state-of-the-art method.
  arXiv  Detail & Related papers  (2026-07-18T16:27:05Z)
• Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs [0.5352699766206807]
  We study a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo.<n>We find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective $0.57 per million tokens.
  arXiv  Detail & Related papers  (2026-07-13T07:04:49Z)
• Orchard: An Open-Source Agentic Modeling Framework [124.68499958175111]
  We present Orchard, an open-source framework for scalable agentic modeling.<n>At its core is Orchard Env, a lightweight environment service providing reusable primitives for sandbox lifecycle management.<n>On top of Orchard Env, we build three agentic modeling recipes.
  arXiv  Detail & Related papers  (2026-05-14T16:35:12Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving [0.0]
  LLM serving often relies on multi-model portfolios spanning a 530x cost range.<n>This trade-off is non-stationary; providers revise pricing, model quality can regress silently, and new models must be integrated without downtime.<n>We present an open-source adaptive router built on cost-aware contextual bandits.
  arXiv  Detail & Related papers  (2026-03-31T18:41:53Z)
• Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints [18.52522897906341]
  We study the problem of routing queries to large language models (LLMs)<n>We propose a batch-level, resource-aware routing framework that jointly optimize assignment for each batch.<n> Experiments on two multi-task benchmarks show that improves accuracy by 1-14% on non-robust counterparts.
  arXiv  Detail & Related papers  (2026-03-25T22:24:11Z)
• Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference [0.0]
  "Pyramid MoA" is a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary.<n>We demonstrate that the system introduces negligible latency overhead (+0.82s) and allows for a tunable trade-off between performance and budget.
  arXiv  Detail & Related papers  (2026-02-23T04:47:47Z)
• RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents [91.0187958746262]
  RouteMoA is an efficient mixture-of-agents framework with dynamic routing.<n>It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query.<n>It refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference.
  arXiv  Detail & Related papers  (2026-01-26T04:22:22Z)
• EMAFusion: A Self-Optimizing System for Seamless LLM Selection and Integration [9.019951211182969]
  We introduce EMAFusion, a new framework that self-optimizes for LLM selection and reliable execution for a given query.<n>We find EMAFusion outperforms the best individual models by over 2.6 percentage points (94.3% vs. 91.7%) while being 4X cheaper than the average cost.<n>Our combined routing approach delivers 94.3% accuracy compared to taxonomy-based (88.1%) and learned model predictor-based (91.7%) methods alone.
  arXiv  Detail & Related papers  (2025-04-14T20:04:54Z)
• CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing [74.14816777318033]
  Collaborative Inference with Token-lEvel Routing (CITER) is a framework that enables efficient collaboration between small and large language models.<n>We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation.<n>Our experiments show that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications.
  arXiv  Detail & Related papers  (2025-02-04T03:36:44Z)
• Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing [53.748685766139715]
  Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size. We propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.
  arXiv  Detail & Related papers  (2024-04-22T23:06:42Z)
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models [53.859446823312126]
  SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks.
  arXiv  Detail & Related papers  (2023-10-11T15:56:00Z)
• Mobility operator service capacity sharing contract design to risk-pool against network disruptions [3.0938904602244355]
  We propose a new mechanism to design risk-pooling contracts between operators to mitigate those costs and improve service resilience during disruptions. We formulate a novel two-stage multicommodity flow model to determine the cost savings of a coalition under different disruption scenarios. Using the proposed method, we identify stable cost allocations among four operating agencies that could yield a 66% improvement in overall network performance over not having any risk-pooling contract in place.
  arXiv  Detail & Related papers  (2020-06-25T16:10:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.