Research On Application Of Generative Dialogue System Based On GPT-2

Man-machine talk between persons technology is favored by learners both in high level teaching person and expert operation of making observations and to do with industry applications.These systems have built-in reply sentences,which are based on rules and retrieval,so they respond quickly and are very popular in the industry in the early years.However,limited by the quantity and quality of the answers in the database,the applications of these dialogue systems can only be used in specific scenarios.In recent years,the development of deep learning technology and natural language processing technology can use semi-supervised deep learning framework to further expand dialogue topics and intentions,but there are still some limitations.For example,Transformer itself cannot directly achieve time series autoregression.When the language model GPT-2 uses masked language modeling,it will mask out the following sequence to avoid the model from obtaining the following information,and can only rely on the previous vocabulary to predict the latter vocabulary,so that the language model Modeling capabilities are lacking.The logic of the Transformer decoder used in the native GPT-2 is to use the Beam Search decoding method based on the possibility of maximizing the output.The generated text has problems such as bland language,poor coherence,and semantic repetition.Although BERT can only mask the information at the current moment,the dialogue system applied by this type of model does not use background information other than dialogue,and its generation ability is still limited,and the generated sentences have many problems such as logical confusion and semantic repetition.In response to the above problems,this study attempts to improve the lack of language modeling ability by introducing contextual background information into the model,thereby improving the efficiency and quality of the tasks performed by the language model.The research constructs two models based on GPT-2,called GPT2 DH model（Double Heads）and GPT2 LM model（Language Modeling）.The GPT2 DH model has two stages,which are Language Modeling and Next Sentence Prediction tasks.The GPT2 LM model removes the NSP task based on the GPT2 DH model,and uses the Language Modeling end of GPT2 alone to perform transfer learning of the back-end tasks.At the same time,the GPT2 DH model has a classification design in addition to GPT2 LM.The classification task is used to predict whether the answer sentence generated by the system is a matching answer sentence for the question sentence,which is used to improve the accuracy of the model generated sentence.The research proposes to optimize the text output by GPT-2 dialogue.In the decoding stage,the top-p sampling method is introduced to avoid the problem of flat,incoherent,or repeated loops in the output text of the beam search decoding method.The validity of the two constructed models,namely GPT2 DH and GPT2 LM,is evaluated;objective indicators such as BLEU,ROUGE,and perplexity are used to test and analyze the generated system.Research statistics show that the response evaluation values of the GPT2 DH model and the GPT2 LM model after pre-training on the three corpora are much higher than those of the language model based on information retrieval.The validity evaluation results of the model show that the two language models GPT2 DH and GPT2 LM are 5 times more than the information retrieval model based on the BLUE and ROUGH indicators,and 2 times that of the seq2 seq model.On the perplexity index,it is 1 times that of the seq2 seq model.Compared with the SOTA model BERT in natural language generation,the GPT2 DH and GPT2 LM models are higher than the BERT model in the BLEU index of 1-4orders,indicating that the sentences generated by the model constructed by the research are similar to the real reply sentences.Sex is higher.The third-order index of ROUGE is also slightly higher than that of the BERT model,which shows that the constructed dialogue system has improved the dialogue accuracy and the reduction in the rate of reply sentences compared with the traditional method.Answers to repetitive and logically confusing questions have been improved to some extent.Finally,the illogical and unanswered situations in the sentences generated by the model are excerpted,analyzed and provided with corresponding improvement measures.