Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss
Not specified in the provided content
Abstract
The study investigates the impact of reducing routed experts during prefill on the performance of mixture-of-experts models, particularly focusing on short-answer question answering.
Reality Card
Reducing the number of routed experts during prefill significantly impacts the F1 score, primarily due to the sensitivity of specific prompt tokens.
The loss in F1 score can be attributed to just a few prompt tokens, with sensitive positions accounting for about half and a fifth of the damage in different models.
The findings are limited by the specific configurations of the models tested and may not generalize across all MoE architectures.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.