Research On Semantic Similarity Of Text Based On Unsupervised Contrastive Learning

At present,large-scale pre-trained language models,such as BERT,can be fine-tuned in many downstream tasks to achieve relatively good results.However,in the task of performing text semantic similarity computation,a semantic matching task is often involved.When no candidate sentence pairs are given in advance,it is inefficient to perform similarity computation and requires a large computational overhead when dealing with knowledge bases.The standard way to reduce computation is to encode the text in the knowledge base into a vector using the trained model and then compare the two vectors by similarity distance.To improve the performance of text similarity computation in the above-mentioned semantic matching task,this paper proposes a method that starts from studying a two-tower network structure to optimize the sentence vector generation of BERT by referring to the classical Sentence-BERT model structure.The main work of this paper includes:(1)This paper proposes a construction method of sample pairs in the unsupervised contrast learning approach to better introduce contrast learning into the sentence vector generation of BERT.To address the shortcomings of the currently popular unsupervised Sim CSE model,positive examples are obtained by modifying the input sentences through random punctuation addition while maintaining the sentence semantics.Negative examples are screened based on similarity distance for utterances containing more semantic information to constitute difficult negative examples,enabling the model to compare more adequately.The use of contrast learning after this sample pair modification enables the generation of better quality BERT sentence vectors.(2)This paper proposes a joint BERT-based multi-task learning model that combines contrast learning with other tasks.The model enables the extraction of word-level interaction features from the interactive task into the sentence representation in the twin-tower task using the knowledge distillation strategy of teacher annealing under unsupervised conditions.The model maintains the advantage of similarity matching efficiency in the twin-tower task,allowing the sentence vectors generated by the trained model to contain more semantic information and better represent the sentence vectors.(3)The proposed model has improved the performance compared to each baseline model on three Chinese datasets,after adjusted to the optimal model with several sets of experiments.The sample pair construction approach of the contrast learning model and the joint multi-task learning model proposed in this paper are compared with the native BERT sentence vector and two representative twin-tower models,Sentence-BERT and unsupervised Sim CSE,using different pre-trained models,such as BERT and Ro BERTa,as encoders.The Spearman correlation coefficient is used as the evaluation metric.After comparison,it is well-demonstrated that the optimized model proposed in this paper is efficient.