Amazon researchers propose HOPE, a second-order pruning method for MoE models

Amazon researchers published HOPE (Higher-Order Pruning of Experts), a method for removing experts from mixture-of-experts language models. Existing approaches score each expert on its own and delete the lowest scorers; HOPE instead solves a quadratic program per layer that also accounts for how often pairs of experts are activated together. The post says calibration needs the same single forward pass as other pruning methods, with one to two seconds per layer added to solve the program. Evaluated on Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE had the best average rank among compared methods at 50% pruning, including a 6.1% gain over the next-best method on agentic coding tasks.