Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

• URL: http://arxiv.org/abs/2604.07609v1
• Date: Wed, 08 Apr 2026 21:27:47 GMT
• Title: Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
• Abstract summary: Large Language Model (LLM) inference is rapidly becoming a core datacenter service.<n>Current serving stacks keep the host on the critical path for orchestration and token-level control.<n>We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path.
• Score: 1.1476836276366738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.

Related papers

• GRAB-ANNS: High-Throughput Indexing and Hybrid Search via GPU-Native Bucketing [39.763467046232584]
  We present GRAB-ANNS, a GPU-native graph index for dynamic hybrid search.<n>We show that GRAB-ANNS achieves up to 240.1 times higher query throughput and 12.6 times faster index construction than state-of-the-art CPU-based systems.
  arXiv  Detail & Related papers  (2026-03-31T11:00:10Z)
• ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference [31.69761251861347]
  Large language models encounter critical GPU memory capacity constraints during long-context inference.<n>We propose ScoutAttention, a novel KV cache offloading framework that accelerates LLM inference through collaborative GPU-CPU attention computation.<n>ScoutAttention features a novel layer-ahead CPU pre-computation algorithm, enabling the CPU to initiate attention computation one layer in advance.<n> Experimental results demonstrate that ScoutAttention maintains accuracy within 2.4% of baseline while achieving 2.1x speedup compared to existing offloading methods.
  arXiv  Detail & Related papers  (2026-03-28T05:06:05Z)
• TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition [0.0]
  This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead.<n>We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI)<n>We show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, single-thread performance is a first-order parameter.
  arXiv  Detail & Related papers  (2026-03-12T21:30:07Z)
• ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs [50.29619653515229]
  ArcLight is a lightweight LLM inference architecture designed from the ground up for many-core CPUs.<n>We show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput.
  arXiv  Detail & Related papers  (2026-03-08T19:20:25Z)
• FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems [39.33711841865621]
  FlashInfer-Bench is a framework that connects kernel generation, benchmarking, and deployment.<n>Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, and a public leaderboard.<n>Using FlashInfer-Bench, we evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design.
  arXiv  Detail & Related papers  (2026-01-01T06:18:53Z)
• CLO: Efficient LLM Inference System with CPU-Light KVCache Offloading via Algorithm-System Co-Design [27.03446161229998]
  We propose CLO, a CPU-light KVCache offloading system via algorithm-system co-design.<n>CLO achieves comparable accuracy to state-of-the-art systems, while substantially minimizing CPU overhead.
  arXiv  Detail & Related papers  (2025-11-18T14:03:21Z)
• SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs [5.760049762453579]
  Accelerating large language models with CPUs enables broader AI access at a lower cost and power consumption.<n>We provide a set of open-source customized sparse kernels that can speed up any PyTorch model.<n>We demonstrate for the first time the use of unstructured sparsity in the attention achieving a $1.14 times$ speedup over the current systems.
  arXiv  Detail & Related papers  (2025-02-18T02:26:34Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter [40.616849959987555]
  We introduce a novel mechanism that fine-tunes Large Language Models (LLMs) with adapters of larger size yet memory-efficient. This is achieved by leveraging the inherent activation sparsity in the Feed-Forward Networks (FFNs) of LLMs. We employ a Mixture of Experts (MoE)-like architecture to mitigate unnecessary CPU computations and reduce the communication volume between the GPU and CPU.
  arXiv  Detail & Related papers  (2024-06-07T14:49:22Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.