Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
Not provided in the abstract
Abstract
This paper explores a method for transferring capabilities from a larger donor language model to a smaller recipient model without requiring explicit neuron-wise semantic alignment.
Reality Card
The proposed Activation-Prune-Merge (APM) method improves the accuracy of a smaller language model by effectively utilizing components from a larger model, achieving an overall accuracy increase from 55.5% to 60.6% across 16 benchmarks.
RTE accuracy increased from 64.3% to 82.3%, demonstrating significant performance improvement.
The method's reliance on specific activation-guided extraction may limit its generalizability across different model architectures.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.