Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training

Charafeddine Mouzouni

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 21 functions out of this paper's own repositories and ran 18 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• Cmouzouni/three-phases-moe

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We model Mixture-of-Experts (MoE) token routing as a congestion game with a single effective parameter-the congestion coefficient γ eff -that quantifies the balance-quality tradeoff. Tracking γ eff across training checkpoints of two open-source MoE models-OLMoE-1B-7B (20 checkpoints, with dense sampling in the surge region) and OpenMoE-8B (6 checkpoints)-reveals a three-phase trajectory: a surge phase where the router learns to balance load (γ eff : 14 → 36-39, peaking in the step 30K-40K region), a stabilization phase where experts specialize under steady balance (B 0 : 2.4 → 2.3, steps 100K-400K), and a relaxation phase where the router trades balance for quality as experts differentiate (γ eff : 27 → 9, steps 400K-1.2M). This non-monotone trajectory-invisible to post-hoc analysis of converged models-reveals that early MoE training prioritizes balance while late training prioritizes quality. The theoretical framework is honest about its limits: the single-type equilibrium reduces to temperature-scaled softmax (held-out L 1 : MFG = 0.199 vs. softmax = 0.200). The game is not a better predictor; it reveals what the temperature means and, critically, how that temperature evolves. Annealing checkpoints confirm that the three phases are pretrainingspecific: γ eff is stable during fine-tuning. We complement the dynamics with an effective congestion decomposition (γ eff = γ explicit +γ implicit ), a multi-type extension that improves load prediction via token clustering on all 16 layers (mean: 30%; robust to cluster count K = 2, 4, 8), and scope diagnostics (K/M, ε l ) that characterize where the per-layer model applies. All confidence intervals are from bootstrap resampling over 50 independent text batches. Code and data: https://github.com/Cmouzouni/three-phases-moe.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.04230")
get_code_for_paper("2604.04230")
have("2604.04230")

Connect an agent — have() is free.