| In recent years,text representation technology based on deep learning has made significant progress and plays an important role in the field of natural language processing.In the English environment,the representation method with artificial neural network as the core has been relatively mature,and the multi-granularity representation system with “character,subword,word”as the token has been basically perfected.Through independent or hybrid embedding of tokens of different granularities,the representation model effectively realizes the automatic abstraction and refinement of text features.Among them,the embedding model with characters and subwords as tokens not only solves the problem of out-of-vocabulary words,but also has obvious performance advantages,which is a mainstreaming method in academia and industry,and is also a research hotspot.Due to the differences in language and the characteristics of hieroglyphics,the development of Chinese representation technology is slightly lagging behind,and there are also shortcomings such as incomplete character and subword granularity token system and ineffective solution to the problem of out-of-vocabulary words.In response to these issues,this dissertation focuses on the mining of the intrinsic features of Chinese characters and their deep integration,proposes a sub-character representation model of Chinese under the constraint of small character set,and constructs a Chinese hierarchical embedding system with “Yao,Zi,Ci” as tokens and corresponding word hybrid embedding scheme based on it.The main content and innovations are as follows:1.Aiming at the shortcomings of Chinese token system,a “CHARM” model is proposed to realize the sub-character representation of Chinese under the constraint of small character set.The model provides Chinese characters with mnemonic abbreviations represented by Latin letters,also known as x QMA,and realizes the explicit fusion and reversible mapping of phonetic and shape features through the Yaoci structure.The three-layer token system of “Yao,Zi,Ci” based on x QMA provides a feasible scheme for Chinese embedding at sub-character level(Yao-level),and also reduces the migration cost between Chinese and English models.2.Aiming at the problem of out-of-vocabulary words,with Chinese-character(Zi)embedding as a breakthrough point,a feature fusion mechanism based on x QMA is proposed,also known as HESbo C.Combined with deep learning methods such as convolutional neural network,recurrent neural network,and self-attention mechanism,a hybrid embedding model of“Yao-Zi” is designed to adapt to different tasks,and the practicality test is carried out in text classification,named entity recognition,and general pre-training tasks.Comparative experiments show that the improved models based on HESbo C can effectively solve the problem of out-ofvocabulary words,and the performance has also been greatly improved.Among them,the CHARMNBERT model,which is used for general pre-training tasks,has the best performance in many tests and can provide better quality Zi embeddings for downstream tasks.3.Based on the improvement of multi-granularity hybrid embedding quality,the HESbo C mechanism is extended to the word embedding model.Giving full play to the advantages of subcharacter embedding of x QMA,the multi-granularity features of “Yao,Zi,Ci” are effectively fused,and the pre-trained models Charm2 Vec and Wo Ch BERT based on hybrid word embedding are realized,and the effectiveness and practicability of the above models are verified in the text classification task.The verification results on both the public dataset and the private dataset show that the performance of the model improved by the HESbo C mechanism has a good increase.In particular,the Wo Ch BERT model performs well in comparison with similar models and has strong adaptability and practicality.The above work has made up for the shortcomings of Chinese text representation system in character and sub-word level,effectively solved the problem of Chinese out-of-vocabulary words,and reduces the migration cost between Chinese and English models.For downstream tasks such as text classification,emotional analysis,public opinion analysis,naming entity identification,etc.,the method mentioned in this dissertation can provide more competitive underlying support. |
