98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

• URL: http://arxiv.org/abs/2603.12646v1
• Date: Fri, 13 Mar 2026 04:33:53 GMT
• Title: 98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
• Abstract summary: We present three staged optimizations for the vLLM Semantic Router.<n>A custom Flash Attention operator for ONNX on ROCm reduces attention memory from $O(n2)$ to $O(n)$.<n>Near-streaming body processing with adaptive chunking eliminates serialization overhead.
• Score: 9.457255218406333
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: System-level routers that intercept LLM requests for safety classification, domain routing, and PII detection must be both fast and operationally lightweight: they should add minimal latency to every request, yet not require a dedicated GPU -- an expensive resource better used for LLM inference itself. When the router co-locates on the same GPU as vLLM serving instances, standard attention's $O(n^2)$ memory makes long-context classification (8K--32K tokens) impossible: at 8K tokens, three concurrent classifiers need ${\sim}$4.5\,GB for attention masks alone, far exceeding the memory left by vLLM. We present three staged optimizations for the vLLM Semantic Router, benchmarked on AMD Instinct MI300X, that solve both the latency and the memory problem. \emph{Stage~1}: a custom CK Flash Attention operator for ONNX Runtime on ROCm reduces attention memory from $O(n^2)$ to $O(n)$ and end-to-end (E2E) latency from 4{,}918\,ms to 127\,ms (\textbf{38.7$\times$}), enabling 8K--32K tokens where SDPA OOMs. \emph{Stage~2}: classical NLP prompt compression (TextRank, position weighting, TF-IDF, and novelty scoring) reduces all inputs to ${\sim}$512 tokens without neural inference, capping both latency and GPU memory at a constant regardless of original prompt length (E2E 127$\to$62\,ms, \textbf{2.0$\times$}). \emph{Stage~3}: near-streaming body processing with adaptive chunking and zero-copy JSON eliminates serialization overhead (E2E 62$\to$50\,ms, \textbf{1.2$\times$}). Cumulatively: \textbf{98$\times$} improvement (4{,}918\,ms to 50\,ms), 16K-token routing in 108\,ms, and a total router GPU footprint under 800\,MB -- small enough to share a GPU with LLM serving and removing the need for a dedicated accelerator. Stage~1 targets AMD ROCm (NVIDIA GPUs already have FlashAttention via cuDNN); Stages~2 and~3 are hardware-agnostic.

Related papers

• DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts [32.57334288518084]
  Modern RL post-training methods such as GRPO and DAPO train on $N$ response sequences of $R$ tokens sampled from a shared prompt.<n>Standard FlashAttention replicates all $P$ prompt tokens $N times across both forward and backward passes.<n>We propose textDualAttention, the first Flash kernel variant eliminates shared-prompt replication during training.
  arXiv  Detail & Related papers  (2026-05-14T21:11:32Z)
• SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data [3.1624024957575982]
  SURGE is a streaming encoding system deployed in production to generate embeddings for over 800 million texts across 40,000 logical partitions.<n>On 10M texts with 4 NVIDIA L4768, SURGE delivers 26,413 texts/s -- matching fixed-times throughput using 12.6times less memory.
  arXiv  Detail & Related papers  (2026-05-01T19:51:50Z)
• ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers [11.576817774473398]
  Existing attention accelerators often trade exact softmax semantics, depend on fused Core kernels, or incur sequential depth that limits FP32 throughput on long sequences.<n>We present bfELSA, an algorithmic reformulation of online softmax attention that reserves exact softmax semantics in real arithmetic with a emphprovable $mathcalO(ulog n)$ FP32 relative error bound.<n>Unlike FlashAttention-2/3, which rely on HMMA/GMMA Core instructions and provide no FP32 path, ELSA operates identically on A100s and resource-
  arXiv  Detail & Related papers  (2026-04-26T16:41:30Z)
• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference [12.135271159221178]
  Production vLLM fleets provision every instance for worst-case context.<n>Production vLLM pools dispatch it to one of two vLLM pools.<n> token-budget routing reduces GPU instances by 17-39%.
  arXiv  Detail & Related papers  (2026-03-13T13:06:50Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity [86.71343842875878]
  NVIDIA's 2:4 Sparse Cores deliver 2x throughput but demand strict 50% pruning.<n>Milder $(2N-2):2N$ patterns preserve accuracy yet receive no hardware support.<n>We present SlideSparse, the first system to unlock Sparse Core acceleration.
  arXiv  Detail & Related papers  (2026-03-05T14:49:16Z)
• FlashMoE: Fast Distributed MoE in a Single Kernel [1.866526462692252]
  FlashMoE is a fully GPU-resident MoE operator that fuses expert computation and inter-GPU communication into a single persistent GPU kernel.<n>We show that FlashMoE achieves up to 9x higher GPU utilization, 6x lower latency, 5.7x higher throughput, and 4x better overlap efficiency compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-06-05T06:29:14Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding [43.28712253351293]
  Key-value (KV) cache has emerged as a bottleneck for efficient long-sequence support. TriForce is a hierarchical speculative decoding system that is scalable for long sequence generation. TriForce's robustness is highlighted by its consistently outstanding performance across various temperatures.
  arXiv  Detail & Related papers  (2024-04-18T05:25:54Z)
• DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
  FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
  arXiv  Detail & Related papers  (2023-10-05T03:47:57Z)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness [80.3586155104237]
  FlashAttention is an IO-aware exact attention algorithm for Transformers. It reduces the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip. FlashAttention and block-sparse FlashAttention enable longer context in Transformers.
  arXiv  Detail & Related papers  (2022-05-27T17:53:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.