MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Harshit Khaitan, Steven Li, Bradley Mcdanel, Sruthikesh Surineni

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 10 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• SafeAILab/EAGLE

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixtureof-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verificationtime expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10-30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.16052")
get_code_for_paper("2602.16052")
have("2602.16052")

Connect an agent — have() is free.