Research On Pre-training Language Model For Biomedical Literature Mining

With the application of high-throughput technology to biomedical research,the amount of biomedical literature is increasing exponentially,and it is particularly important to use literature mining technology to automatically extract knowledge from the literature accurately.In recent years,the rise of pre-training language models such as BERT has greatly promoted the development of the research of biomedical literature mining.These works use a deep Transformer-based model structure and specific pre-training tasks to perform pre-training on a large-scale general-purpose corpus,which effectively improves the language representation ability of pre-trained language models.Also,they show excellent entity recognition performance in the biomedical named entity recognition task through fine-tuning.However,on the one hand,the existing works which transfer pre-trained language models to the biomedical field is only achieved by replacing the corpus,and there is much room for improvement.On the other hand,the models for biomedical named entity recognition models based on pre-trained language models disadvantage of too large scale and too slow speed.In response to the above two issues,this article makes the following research of biomedical pre-training language model.First,considering the characteristics of the biomedical literature,we propose two biomedical pre-training tasks to solve the problem of how to introduce external biomedical domain knowledge into the pre-training language model,which are the biomedical entity-level masking language model task named BEMLM and the biomedical entity type prediction task named BETP.In detail,the biomedical entities are automatically extracted through the named entity recognition model,and the extraction results are used to complete the pre-training tasks.And through the above pre-training tasks,we propose a BERT-based biomedical pre-training language model named BioTypeBERT,which combines with BEMLM and BETP for joint learning by establishing a corresponding classifier in the prediction layer.It solves the lack of existing research that does not introduce external biomedical domain knowledge,and has certain guiding significance for the development of biomedical pre-training language models.And after pre-training,BioTypeBERT is used in the application of various biomedical literature mining tasks by fine tuning.Finally,BioTypeBERT was verified experimentally through the five public data sets.And for how to efficiently compress biomedical named entity recognition models based on pre-trained language models,we propose a knowledge distillation method based on dynamic weights named DKG,which solves the problem that the existing knowledge distillation method relies too much on experience for parameter adjustment.DKG uses dynamic weight functions to simulate nonlinear learning curves and optimize the knowledge distillation process for efficient compression of pre-trained language models.It has certain reference significance for model compression algorithm.And we propose a biomedical named entity recognition model named FastBioNER based on DKG.In detail,the pre-trained language model is compressed as a teacher model to a smaller student model after fine-tuning.Finally,FastBioNER was experimentally verified through the public data sets.