MoE Competition Documentation

This site gives competition participants the technical background needed to reason about Mixture-of-Experts (MoE) load balancing, with a focus on DeepSeek-style sparse expert models.

The competition simulator evaluates dynamic expert placement policies using trace-derived expert hotness. These notes explain why expert routing creates load imbalance, how DeepSeek-style MoE models route tokens, and what the simulator metrics are intended to capture.

Contents

• DeepSeek Overview
  • Why DeepSeek Matters For MoE Load Balancing
  • Architecture Lineage
  • Architecture Components
  • Practical Interpretation
• Introduction To Transformers
  • Tokens And Hidden States
  • Self-Attention
  • Multi-Head Attention
  • Feed-Forward Networks
  • Residual Connections And Normalization
  • Autoregressive Inference
  • Why Transformers Lead To MoE
• Mixture-of-Experts Documentation
  • Dense FFN Versus MoE FFN
  • Core Terms
  • Why MoE Models Are Efficient
  • Why MoE Models Are Hard To Serve
  • Load Balancing Mechanisms
  • Competition Metrics
• DeepSeek MoE Routing Algorithm
  • Router Inputs And Outputs
  • DeepSeekMoE Design
  • DeepSeek-V3 Routing Path
  • Pseudocode
  • Group-Limited Routing
  • Auxiliary-Loss-Free Load Balancing
  • Relation To The Competition Simulator
• Competition Context
  • Simulator Inputs
  • Placement Output
  • Evaluation Loop
  • What Makes A Good Policy
  • How To Read PAR
  • How To Read Transmit Amount
  • Composite Score
• Simulator And Reference Submissions
  • Repository Map
  • Submission API
  • Smoke Submission
  • Hot-Expert Baseline Submission
  • DeepSeek EPLB Walkthrough
• References