The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing

• URL: http://arxiv.org/abs/2606.23969v1
• Date: Mon, 22 Jun 2026 21:48:53 GMT
• Title: The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing
• Abstract summary: GPU-CC now preserves GPU-local performance on NVIDIA B300, BF16 matmul runs at 0.998x of non-confidential performance.<n>Yet serving under Intel TDX plus GPU-CC still loses 13-27% of throughput, and KV-cache restore latency can more than double.<n>Bridge model explains a +131% KV-restore penalty and a 34x model slowdown.
• Score: 6.69555823710995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: GPU Confidential Computing (GPU-CC) now preserves GPU-local performance: on NVIDIA B300, BF16 matmul runs at 0.998x of non-confidential performance. Yet LLM serving under Intel TDX plus GPU-CC still loses 13-27% of throughput, and KV-cache restore latency can more than double. This paper studies that gap on two Blackwell platforms, RTX Pro 6000 and B300 HGX, and identifies its dominant cause: the confidential VM-GPU bridge, not GPU compute. We find that GPU-CC turns host/device movement into a serialized, high-setup-cost channel. Secure copies do not gain CUDA-stream concurrency within a context, asynchronous transfers block at the runtime boundary, and small crossings pay a fixed toll. This violates the assumptions of modern inference runtimes, where DMA is expected to be cheap, concurrent, and asynchronous. In vLLM dense decode, the gap closes around 44x-slower small alloc-and-copy operations; targeted patches reject alternative explanations. A scheduling flag recovers 57% of the gap, while a worker-thread drain recovers up to 92% in qualified high-concurrency runs. The same bridge model explains a +131% KV-restore penalty and a 34x model-load slowdown. Blackwell also changes the confidential tenancy unit. We qualify confidential multi-GPU NVSwitch tenants on B300, including 510 GB/s NVLink P2P inside a CVM and concurrent isolated tenants, and identify the remaining fabric-attestation gap for production confidential AI platforms.

Related papers

• Benchmarking Confidential Computing Performance on NVIDIA Blackwell GPUs [0.0]
  This paper measures the performance impact of running large language model inference and training inside a Trusted Execution Environment.<n>The performance impact is derived from paired confidential versus non-confidential runs on a single physical host.<n>We localize each cost to a specific encrypted boundary, give a microbenchmark that predicts the serving penalty to within a submission count, and end with concrete deployment guidance.
  arXiv  Detail & Related papers  (2026-08-27T03:37:35Z)
• A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi [0.0]
  This report keeps the hardware and asks what a model that fits can do.<n>We deploy Mini-V-4.6, a modern multimodal assistant pairing a SigLIP2 encoder and window-attention merger.<n>The system answers an image question end-to-end in 1.7s.
  arXiv  Detail & Related papers  (2026-07-16T04:48:44Z)
• Scaling Behaviors of Evolutionary Algorithms on GPUs: When Does Parallelism Pay Off? [43.96509049196842]
  Evolutionary algorithms (EAs) are increasingly implemented on graphics processing units (GPUs) to leverage parallel processing capabilities for enhanced efficiency.<n>We investigate how GPU parallelism alters the behavior of EAs beyond simple acceleration metrics.<n>Our results reveal that the impact of GPU acceleration is highly heterogeneous and depends strongly on algorithmic structure.
  arXiv  Detail & Related papers  (2026-01-26T12:55:21Z)
• LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs [45.51664355320938]
  We present an end-to-end/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on commodity GPUs.<n>This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%.
  arXiv  Detail & Related papers  (2025-12-17T10:51:45Z)
• GPU-Accelerated Interpretable Generalization for Rapid Cyberattack Detection and Forensics [0.0]
  IG mechanism recently published in IEEE Transactions on Information Forensics and Security delivers state-of-the-art, evidence-based intrusion detection.<n>We present IG-GPU, a PyTorch re-architecture that offloads all pairwise intersections and subset evaluations to commodity GPU.<n>In 15k-record NSL-KDD dataset, IG-GPU shows a 116-fold speed-up over the multi-core CPU implementation of IG.
  arXiv  Detail & Related papers  (2025-07-16T12:38:19Z)
• Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving [4.309392302169281]
  Engine-level prefill-decode (PD) disaggregation avoids interference but incurs higher hardware and coordination overhead.<n>PD achieves up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM; outperforms SG by up to 2x; and matches or exceeds disaggregated vLLM.
  arXiv  Detail & Related papers  (2025-07-09T07:27:18Z)
• Minute-Long Videos with Dual Parallelisms [57.22737565366549]
  Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos.<n>We propose a novel distributed inference strategy, termed DualParal.<n>Instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs.
  arXiv  Detail & Related papers  (2025-05-27T11:55:22Z)
• HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading [79.38548165722229]
  HEADINFER offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU.<n>We demonstrate HEADINFER maintains computational efficiency while significantly reducing memory footprint.
  arXiv  Detail & Related papers  (2025-02-18T06:26:05Z)
• NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference [22.24693513075852]
  NEO is an online LLM inference system that offloads part of attention compute and KV cache states from the GPU to the local host CPU. We present NEO, an online LLM inference system that offloads part of attention compute and KV cache states from the GPU to the local host CPU.
  arXiv  Detail & Related papers  (2024-11-02T05:15:44Z)
• Endor: Hardware-Friendly Sparse Format for Offloaded LLM Inference [47.043257902725294]
  We propose a novel sparse format that compresses unstructured sparse pattern of pruned LLM weights to non-zero values with high compression ratio and low decompression overhead. Compared to offloaded inference using the popular Huggingface Accelerate, applying Endor accelerates OPT-66B by 1.70x and Llama2-70B by 1.78x.
  arXiv  Detail & Related papers  (2024-06-17T15:55:08Z)
• DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training [13.953918004371493]
  DisCo-CLIP is a memory-efficient CLIP training approach. DisCo-CLIP can enable contrastive training of a ViT-B/32 model with a batch size of 32K or 196K.
  arXiv  Detail & Related papers  (2023-04-17T17:58:21Z)
• PLSSVM: A (multi-)GPGPU-accelerated Least Squares Support Vector Machine [68.8204255655161]
  Support Vector Machines (SVMs) are widely used in machine learning. However, even modern and optimized implementations do not scale well for large non-trivial dense data sets on cutting-edge hardware. PLSSVM can be used as a drop-in replacement for an LVM.
  arXiv  Detail & Related papers  (2022-02-25T13:24:23Z)
• CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU [8.633428365391666]
  CryptGPU is a system for privacy-preserving machine learning that implements all operations on the GPU. We introduce a new interface to embed cryptographic operations over secret-shared values into floating-point operations. We show that our protocols achieve a 2x to 8x improvement in private inference and a 6x to 36x improvement for private training.
  arXiv  Detail & Related papers  (2021-04-22T09:21:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.