Higher-Order Pruning of Experts in Mixture-of-Experts Language Models

The article discusses the introduction of HOPE (Higher-Order Pruning of Experts), a second-order pruning method for Mixture-of-Experts (MoE) models that accounts for pairwise expert interactions. This method improves upon existing first-order pruning techniques by minimizing pruning error through a more comprehensive evaluation of expert cooperation. HOPE has demonstrated gains of up to +6.1% on agentic coding tasks compared to the next-best pruning method.