Your Experts Have Partners. There's HOPE for Keeping Them Together

Why it matters

This article, authored by the researchers behind the method, explains HOPE, a second-order approach to pruning experts in Mixture-of-Experts language models. MoE architectures keep every expert network resident in memory even though only a small fraction activates for any given token, and in Qwen3.5-122B the expert MLPs dominate the model's memory footprint. Pruning experts directly addresses this bottleneck without training and without the latency drawbacks of dynamically loading experts. The key insight is that existing pruning techniques score each expert in isolation, ignoring the stable cooperative groups that form among experts during training. Deleting one member of such a group can silently break the contributions of its partners, an error that compounds as pruning becomes aggressive. HOPE instead models pairwise expert interactions through a quadratic program built from calibration data, capturing both individual importance and cooperative contribution. The authors report that HOPE achieves the best average rank among pruning methods and delivers notable gains on agentic coding benchmarks, and they provide guidance for applying it to other models.