Design And Implementation Of Chat System Based On Python Crawler And NLP

In the context of increasing life pressure caused by fierce social competition,designing an intelligent chat system can effectively alleviate this phenomenon.According to the different implementation methods of chat systems,they can be divided into retrieval based chat systems and generative based chat systems.The response statements of the retrieval model need to be retrieved and matched from the corpus,and the generated response statements have a smooth structure,but the scope of the response is limited.The statements generated by the generative model are not limited by the scope of the corpus,but the amount of information generated is usually small.Someone has proposed a combination of the two methods,but it cannot guarantee the high quality of the final reply statement.If the human-computer conversation model is not well designed,the user’s sense of experience will be greatly affected in actual chat system applications.Aiming at the above problems,this paper designs and implements an intelligent chat system based on Python crawler and NLP.Firstly,it uses bidirectional attention and random proportional mask mechanism to improve the machine translation BERT model of the chat system;Then,by introducing background knowledge and adding a general corpus,the combination method of retrieving and generating conversation models in chat systems is improved.The specific content is as follows:(1)A machine translation BERT model incorporating proportional allocation mask and coding fusion is proposed.Firstly,the semantic relationship between two sentences in the input is calculated to generate an alignment matrix.The convolutional neural network is used to extract the features of the alignment matrix and construct a sentence dimension code for the input layer.The bidirectional attention fusion word dimension code and sentence dimension code are used to enrich the coding information of the input sentence;Of the 15%of words masked out,half are random,and the other half are based on the proportion of sentence components.By adding a recurrent neural network to the encoder of the model to analyze the key parts of the input sentence,the key parts are masked down by 60%to improve the prediction ability of the model.(2)A fusion model of DAM and GPT-2 based on background knowledge and general corpus is proposed.Based on the retrieval dialog model DAM,cosine similarity is used to select relevant background knowledge from the knowledge base,and attention mechanism is used to fuse background knowledge,input statements,and candidate statements,respectively,to improve the accuracy of retrieval dialog model responses;Improve the combination of the two conversation models by using crawler technology to add a generic corpus and set thresholds,calculate the correlation between input and output statements,and compare them with the threshold.If it is below the threshold,search for and match the relevant output from the generic corpus.(3)Based on the above work,an intelligent chat system is designed and implemented.The front end Vue framework is used to create chat system pages,and the back end Flask framework is used to implement corresponding functions.Combined with a machine translation model and a search generation model,user information and chat records are stored in a MySQL database.The Redis database is used to cache cookies and other data.According to the B/S architecture,the whole chat system is achieved.