Research On Entity Relationship Extraction Based On Self-Attention And Knowledge Distillation

Entity relationship extraction is one of the basic tasks in the field of natural language processing.Its purpose is to predict the relationship type of a given entity in the text,and extract the triple information in the form of<subject,relationship,object>from the text.It plays an important role in the construction of knowledge graphs,intelligent search and automatic question answering.In text sentences,there is a certain relationship between words.When extracting entity relationships,effectively using the correlation between them can help improve the effect of entity relationship extraction.Therefore,this paper introduces the self-attention mechanism into the CASREL model to fully capture the correlation between words in the sentence and improve the entity relationship extraction performance of the model.On the other hand,considering that the model will occupy more GPU resources and take a long time during training.Therefore,this paper adopts the method of knowledge distillation to transfer the knowledge learned by the large model to another lightweight simple model,so that the simple model can learn the language representation ability in a short period of time under the condition of limited resources.This paper studies the performance improvement of the entity relationship extraction model and the optimization of resources occupied by model training.The main work is as follows:（1）Aiming at the problem that the relationship between the words in the sentence in the text is not effectively used in the entity relationship extraction,this paper proposes an entity relationship extraction model CASREL＿Att based on the CASREL fusion self-attention mechanism.After predicting the subject in the sentence,the model effectively uses the correlation between the subject,the object and the relationship type in the sentence,and avoids the interference of objects and relationship types that are not related to the subject when predicting the object and relationship type in the next step.Thereby improving the accuracy of entity relationship extraction and improving model performance.The experimental results show that on the Web NLG dataset and the NYT dataset,the F₁ index of the CASREL＿Att model is 0.6%and 2.4%higher than that of the CASREL model,respectively,indicating that the CASREL＿Att model proposed in this paper effectively improves the performance of entity relationship extraction;（2）Aiming at the problem that the entity relationship extraction model occupies a lot of hardware resources and takes a long time to train,this paper proposes the Distill＿CASREL model based on knowledge distillation.The CASREL（LSTM）model is used as the student model,the CASREL＿Att model is used as the teacher model,and the teacher model is used to guide the student model for training.During the training process,the student model learns how to generalize to new data,thereby performing knowledge distillation on the model and obtaining a new lightweight high-performance model.The experimental results show that on the Web NLG dataset and the NYT dataset,the F₁ index of the Distill＿CASREL model is 1.2%and 1.5%higher than that of the CASREL model,the GPU memory usage is reduced by71.6%,and the training time is shortened by 65.9%.After knowledge distillation,the Distill＿CASREL model effectively optimizes the GPU memory usage during model training and shortens the training time while ensuring performance.