Relation Extraction Based On Gated Transformer And Type Attention

With the rapid development of the Internet and the rapid growth of unstructured text data,how to locate keywords efficiently and accurately has become an important research direction for us to explore.As the main research direction of information extraction,relation extraction is playing a huge role in knowledge graphs and web search.Thanks to the heuristic distant supervision method,distant supervision relationship extraction can automatically identify entity-pair relationships.With the rapid development of machine learning and deep learning,relation extraction uses convolutional neural networks to improve the accuracy of prediction.Nevertheless,the existing models still have the following problems: Convolutional neural network can extract short-distance information of text words,but cannot obtain long-distance dependent information in the context;Transformer can effectively integrate global semantic information and local semantic information,but ignores the relationship with the average semantics in the context;Even if different entities have the same type,the model should use different weights to represent the entity type,emphasizing the contribution of important types to the current entity.In response to the above-mentioned shortcomings,this thesis has contributed the following innovations:(1)Propose a model that integrates linear attenuation entities and gated TransformersIn addition to using the traditional word vector and position vector as the input vector,the model also adds a linear attenuation vector to each word,so that each input vector considers the word/position vector while taking into account the relationship between each word and entity pair.The linear attenuation relationship is negatively related to the distance between the word and the entity pair.While using Transformer to effectively combine global and local semantic relations,the concept of gating is introduced,making Transformer also pay attention to average semantic information.This model uses sentence-level and bag-level attention mechanisms to automatically reduce the impact of incorrect data on the accuracy of the model.The experimental results on the general data set show that the model can perform relationship classification better,and the accuracy and AUC value are better than most models using Transformer.(2)Proposed an entity type model based on dynamic weightThe model introduces additional prior knowledge,and extracts the unique feature vector of each entity through the attention mechanism of dynamic weights.The type of entity can effectively reduce the scope of the relationship extraction results,but treating each type equally does not give the greatest effect of the entity type.The model uses the attention mechanism to dynamically calculate the weight of each type to obtain the type feature vector corresponding to each entity.According to the experimental results,we can conclude that the entity type based on dynamic weight can further improve the AUC value of the model.