ComplianceGate: Classifier-Gated Multi-Tier LLM Routing for Inference in Regulated Industries

• URL: http://arxiv.org/abs/2606.31163v2
• Date: Wed, 01 Jul 2026 08:21:43 GMT
• Title: ComplianceGate: Classifier-Gated Multi-Tier LLM Routing for Inference in Regulated Industries
• Abstract summary: PII in user queries can reach model endpoints before the system determines whether that data should leave its jurisdictional boundary.<n>We propose a classifier-gated routing architecture that enforces compliance by design.<n>Simple queries reach small, fast models at a fraction of the cost.
• Score: 0.609170287691728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models deployed in regulated industries operate under two constraints: compliance enforcement and cost efficiency. Personally identifiable information (PII) in user queries can reach model endpoints before the system determines whether that data should leave its jurisdictional boundary. Serving all queries through a single large model consumes full GPU capacity regardless of query complexity while offering no mechanism for geographic routing. Mixture-of-Experts architectures do not address this routing occurs between expert layers within the model after data has already arrived at the endpoint, with all experts loaded in memory regardless of query complexity. We propose a classifier-gated routing architecture that enforces compliance by design. A trained encoder classifier sits before any decoder inference, evaluating each query for complexity and data sensitivity, then routing it to an appropriately sized dense model in the appropriate geographic location. PII-containing queries route to local endpoints before any LLM computation begins, making data residency violations structurally impossible. Simple queries reach small, fast models at a fraction of the cost. Our evaluation on 600 queries demonstrates 39% median latency reduction, 33-52% cost savings depending on query distribution, and generation throughput of 122-200 tokens/second versus 50-64 for the baseline. The encoder classifier achieves 99.2% accuracy with near-perfect PII recall at 7ms inference overhead, establishing pre-inference classification as a practical path to compliance-by-design LLM deployment.

Related papers

• Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems [7.440012812118169]
  RAG systems commonly retrieve a fixed number of documents to ground generation.<n>simple queries suffer over-retrieval (adding noise and cost) while complex queries are under-retrieved.<n>We propose a practical, general framework for query-adaptive retrieval depth.
  arXiv  Detail & Related papers  (2026-09-11T19:57:12Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarks [1.5229257192293202]
  We introduce compute-grounded reasoning (CGR) for spatial-aware research agents.<n>CGR is a design paradigm for spatial-aware research agents in which every answerable sub-problem is resolved by deterministic computation.<n>We show that CGR yields competitive accuracy while maintaining interpretability.
  arXiv  Detail & Related papers  (2026-04-13T22:22:07Z)
• Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation [0.0]
  Large Language Models (LLMs) exhibit a well-documented positional bias when processing long input contexts.<n>This limits knowledge-retrieval applications that embed large structured knowledge bases directly in the context.<n>We propose Self-Describing Structured Retrieval (SDSR), a lightweight framework in which structured data files embed human-authored navigational metadata at the file's primacy position.
  arXiv  Detail & Related papers  (2026-03-28T14:12:43Z)
• Agentic Multi-Source Grounding for Enhanced Query Intent Understanding: A DoorDash Case Study [0.8924669503280332]
  We introduce an Agentic Multi-Source Grounded system that addresses both failure modes.<n>Rather than predicting a single label, the model emits an ordered multi-intent set.<n>The system is deployed in production, serving over 95% of daily search impressions.
  arXiv  Detail & Related papers  (2026-03-02T05:51:05Z)
• From HNSW to Information-Theoretic Binarization: Rethinking the Architecture of Scalable Vector Search [0.7804710977378487]
  This paper analyzes the architectural limitations of the dominant "HNSW + float32 + cosine similarity" stack.<n>We introduce and empirically evaluate an alternative information-theoretic architecture based on maximally informative binarization (MIB)<n>Results demonstrate retrieval quality comparable to full-precision systems, while achieving substantially lower latency and maintaining constant throughput at high request rates.
  arXiv  Detail & Related papers  (2025-12-16T23:24:37Z)
• Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation [55.47971671635531]
  Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA)<n>Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge.<n>Existing systems primarily rely on unstructured documents, while largely overlooking relational databases.
  arXiv  Detail & Related papers  (2025-09-30T22:19:44Z)
• Leveraging Generative Models for Real-Time Query-Driven Text Summarization in Large-Scale Web Search [54.987957691350665]
  Query-Driven Text Summarization (QDTS) aims to generate concise and informative summaries from textual documents based on a given query.<n>Traditional extractive summarization models, based primarily on ranking candidate summary segments, have been the dominant approach in industrial applications.<n>We propose a novel framework to pioneer the application of generative models to address real-time QDTS in industrial web search.
  arXiv  Detail & Related papers  (2025-08-28T08:51:51Z)
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity [59.57065228857247]
  Retrieval-augmented Large Language Models (LLMs) have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA) We propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs based on the query complexity. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems.
  arXiv  Detail & Related papers  (2024-03-21T13:52:30Z)
• SQLNet: Scale-Modulated Query and Localization Network for Few-Shot Class-Agnostic Counting [67.97870844244187]
  The class-agnostic counting (CAC) task has recently been proposed to solve the problem of counting all objects of an arbitrary class with several exemplars given in the input image.<n>We propose a novel localization-based CAC approach, termed Scale-modulated Query and Localization Network (Net)<n>It fully explores the scales of exemplars in both the query and localization stages and achieves effective counting by accurately locating each object and predicting its approximate size.
  arXiv  Detail & Related papers  (2023-11-16T16:50:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.