Research And Application Of Transformer Language Model Based On Semantic Completeness

In recent years,natural language processing(NLP)has become an important research direction in the field of artificial intelligence.Among them,Transformer-based pre-trained language models have attracted a lot of attention from scholars.The Transformer model represents the one-to-many relationships between individual tokens and the rest of the tokens using self-attention mechanisms.It generates word embeddings through training to better capture the semantics of the text.However,existing Transformer-based language models primarily focus on word-level representations and fail to explore semantic representations at higher levels such as sentences,paragraphs,and documents.To address this issue,this paper proposes the concept of Semantic Completeness(SC)feature and applies it to named entity recognition and relation extraction to enhance the features,thereby improving the performance of downstream tasks.The specific research contents of this paper are as follows:(1)Proposed a named entity recognition method based on semantic completeness.Building upon the existing language models for measuring semantic similarity,the concept of semantic completeness is introduced to represent the relationship between sentences.Based on the theory of semantic completeness,a Transformer-based model called SCM-E(Semantic Completeness Model-Entity)is developed to improve the fragment-based named entity recognition model.The SCM-E model utilizes the masked contrastive method to construct semantic completeness representation vectors and introduces a new pre-training task called Sim-WWM(Similarity-Whole Word Mask)on top of BERT to enhance the model’s understanding of entities.Experimental results demonstrate significant improvements of the SCM-E model on multiple datasets,particularly a 3.15%enhancement over the baseline model on the ADE dataset.(2)An approach for relation extraction based on semantic completeness is proposed.To address the issue of text dependencies in relation classification,a new analysis method is introduced,which divides the text on which relation classification relies into Si and So.Based on this,the contribution of Si and So to relation classification is analyzed using the theory of semantic completeness,and an optimized Semantic Completeness Model-Relation(SCM-R)model is proposed.The model utilizes the Transformer framework and combines the semantic completeness vectors with Si and So to construct relation dependency vectors that assist in relation classification.Experimental results demonstrate that the SCM-R model performs exceptionally well on multiple datasets,with the highest improvement of 5.40%compared to the baseline model.(3)Constructed a multi-task NLP platform.Building upon the previous research,the platform integrates named entity recognition,masked prediction,and semantic similarity calculation tasks into a unified platform,with expandable space for adding more tasks.The platform provides visualizations of the results for different tasks,making the user interaction process more user-friendly.