Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

• URL: http://arxiv.org/abs/2608.08730v2
• Date: Tue, 11 Aug 2026 13:35:56 GMT
• Title: Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference
• Abstract summary: We introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost.<n>We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet the overhead of WebGPU per-operation dispatch remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. Therefore, we conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.

Related papers

• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• Performance of Small Language Model Pretraining on FABRIC: An Empirical Study [2.2070336216767763]
  In this work, we investigate the performance of pretraining techniques for smaller-sized LLMs on an experimental testbed available to academic users at no charge.<n>We use GPT-2 medium and large models and pretrained them using open-source packages, namely, Alpa and Ray.<n>We observed that Alpa's execution plans that collectively optimized intra-operator and inter-operator/pipeline parallelism consistently performed the best when GPUs were geographically distributed.
  arXiv  Detail & Related papers  (2026-02-02T17:58:47Z)
• Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models [8.341777627286621]
  Large language models have demonstrated extraordinary performance in many AI tasks but are expensive to use, even after training, due to their requirement of high-end GPU.<n>Recently, a distributed system called PETALS was developed to lower the barrier for deploying LLMs by splitting the model blocks across multiple servers with low-end GPU distributed over the Internet.<n>We present the first systematic study of the resource allocation problem in distributed LLM inference, with focus on two important decisions: block placement and request routing.
  arXiv  Detail & Related papers  (2025-12-26T06:13:59Z)
• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective [6.51239603014107]
  Large Language Models (LLMs) have pushed training workloads beyond the limits of single-node analysis.<n>We present a comprehensive characterization of LLM training across diverse real-world workloads and hardware platforms.
  arXiv  Detail & Related papers  (2025-09-12T16:05:07Z)
• Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving [4.309392302169281]
  Engine-level prefill-decode (PD) disaggregation avoids interference but incurs higher hardware and coordination overhead.<n>PD achieves up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM; outperforms SG by up to 2x; and matches or exceeds disaggregated vLLM.
  arXiv  Detail & Related papers  (2025-07-09T07:27:18Z)
• Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computing [67.98609858326951]
  Intra-DP is a high-performance collaborative inference system optimized for deep neural networks (DNNs) on mobile devices.<n>It reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.<n>The evaluation demonstrates that Intra-DP reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-07-08T09:50:57Z)
• Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study [11.306063471976369]
  We benchmark the overhead introduced by TEE mode across various large language model (LLM) inference tasks. Our results indicate that while there is minimal computational overhead within the GPU, the overall performance penalty is primarily attributable to data transfer.
  arXiv  Detail & Related papers  (2024-09-06T02:44:27Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• Distributed Inference and Fine-tuning of Large Language Models Over The Internet [91.00270820533272]
  Large language models (LLMs) are useful in many NLP tasks and become more capable with size. These models require high-end hardware, making them inaccessible to most researchers. We develop fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput.
  arXiv  Detail & Related papers  (2023-12-13T18:52:49Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer [63.25665813125223]
  We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation. It achieves better trade-off between performance and efficiency than CNN-based models. Experiments on mainstream benchmarks demonstrate the effectiveness of our proposed RTFormer.
  arXiv  Detail & Related papers  (2022-10-13T16:03:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.