| Massively pre-trained language models such as BERT are language models trained with a large amount of corpus.They have gained tremendous success in many downstream NLP tasks.However,they are computationally expensive to fine-tune,slow for inference,and have large storage requirements.So transfer learning with adapter modules has been introduced and has become a great solution for those problems.Inserting adapter modules in pre-trained language model achieves comparable performance to full fine-tuning on most NLP tasks,without the need of fine-tuning the whole model for a downstream task.The latest AdapterFusion model merge multiple adapters to incorporate knowledge from different tasks.Nevertheless,recent studies reveal that the parameters in adapters are actually still quite redundant,which could slow down inference speed when fusing multiple adapters for a specific downstream task,and thus can be further reduced.Our research mainly focuses on reducing the redundant parameters in adapters and improving the inference speed of AdapterFusion.The main contributions are summarized as follows:1)We propose three novel ways to prune the adapter modules iteratively based on the prestigious Lottery Ticket Hypothesis which are weight pruning,neuron pruning,and adapter module pruning.Extensive experiments show that the pruned adapters can achieve state-of-the-art results,with sizes reduced to 21%while performance remains unchanged.Some pruned adapters even outperform those with the same size fine-tuned alone without pruning.2)We further implement the transfer learning between tasks by sharing the winning ticket found by the proposed approaches,with results justifying that the transferred model can perform better.3)Pruned AdapterFusion model with our scheme can achieve state-of-the-art results,reducing almost half of size while keeping performance intact and speeding up 40%in model inference.We propose a new indicator LIA(Layer Influence Of Adapter),to quantify the utilization of adapters at each layer and identify the most influential adapters in the model. |
