Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

Jędrzej Maczan

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 10 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• jmaczan/torch-webgpu

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

WebGPU's security-focused design imposes per-operation validation that compounds across the many small dispatches in neural network inference, yet the true cost of this overhead is poorly characterized. We present a systematic characterization of WebGPU dispatch overhead for LLM inference at batch size 1, spanning four GPU vendors (NVIDIA, AMD, Apple, Intel), two native implementations (Dawn, wgpu-native) and three browsers (Chrome, Safari, Firefox), and two model sizes (Qwen2.5-0.5B and 1.5B). Our primary contribution is a sequential-dispatch methodology that reveals naive single-operation benchmarks overestimate dispatch cost by ∼20×. The true per-dispatch cost of WebGPU API overhead alone is 24-36 µs on Vulkan and 32-71 µs on Metal, while the total per-operation overhead including Python/framework cost is ∼95 µs, which turns out to be a distinction critical for optimization. On Vulkan, kernel fusion reduces dispatches from 876 to 564, improving throughput by 53%, while CUDA fusion provides no benefit, confirming that per-operation overhead is a primary differentiator. End-to-end LLM inference was tested across three major operating systems (Linux, Windows, macOS) with five different backends (CUDA, MPS, CPU, WebGPU/Dawn, browser/WebLLM). We built torch-webgpu, our own PrivateUse1-based out-of-tree PyTorch backend and an FX-to-WebGPU compiler, which on our reference platform (NVIDIA RTX 5090/Dawn/Vulkan, float32), achieves 21.0 tok/s (0.5B) and 17.9 tok/s (1.5B), which corresponds to 11-12% of CUDA performance. At dtype-matched float32, a mobile CUDA GPU (RTX PRO 2000) achieves 1.4× WebGPU's throughput despite ∼6× less compute than RTX 5090. For dispatch overhead, backend choice (Vulkan vs. Metal) is the dominant factor, although implementation choice also matters substantially within a backend (2.2× for Metal). In terms of dispatch vs kernel compute efficiency, we conclude that at batch=1 with the current dispatch-heavy pipeline, per-operation overhead dominates regardless of kernel quality. All code, benchmarks, and raw data are open source.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.02344")
get_code_for_paper("2604.02344")
have("2604.02344")

Connect an agent — have() is free.