Design And Implementation Of Unsupervised Dialogue Structure Mining System

Task-based dialogue systems have been widely used in various fields of the Internet,but there are two main problems in the process of building,maintaining and updating task-based dialogue systems.First,building a task-based human-machine dialogue system requires pre-defined dialogue tasks.It requires human experts to read a large number of primitive human-human dialogue materials in specific business scenarios,fully understand and extract dialogue structure information,and carefully design dialogue tasks,which is time-consuming and labor-intensive;Second,the human-human dialogue corpus required for training task-based dialogue systems need to be manually annotated in detail.With the continuous emergence of new fields,the cost of labeling is high and the timeliness is poor.The above reasons make it difficult for task-based dialogue systems to update and iterate in time.Therefore,if the information needed to construct the dialogue system can be found automatically and unsupervised from the large-scale unlabeled human-human dialogue data to assist the construction and subsequent updating of the dialogue system,it will greatly improve the construction efficiency of the dialogue system and reduce costs.In response to the above problems,this thesis conducts work on the basis of a systematic review of existing related research,including the following:According to different characteristics of dialogue data,two dialogue structure mining algorithms based on unsupervised learning method are proposed.One is a dialogue structure mining algorithm based on dependency syntactic parsing.The algorithm uses named entity recognition,dependency syntactic parsing and other technologies to mine the inter-word dependencies,infers the dialogue structure information from these inter-word relationships,this algorithms can completes simpler types of dialogue structure mining and dialogue task definition;The second is a dialogue structure mining algorithm based on cluster analysis.The algorithm first uses the BERT pre-training model to fuse the context information of the slot value,and then uses the clustering algorithm to cluster,and obtains the dialogue structure information through the cluster information.This algorithm can complete the dialogue structure mining and dialogue task definition in the dialogue with less data.Based on the structure information obtained by the above algorithm,the transfer relationship between the dialogue structure information is further excavated,and the dialogue structure chart is obtained to assist the construction of dialogue strategies.Finally,based on the software engineering method,a dialogue structure mining demonstration system for large-scale human-human dialogue data is designed and implemented.The function and performance of the system are verified by tests to meet the requirements,besides,this system can accomplish the goal of dialogue structure mining and achieve the purpose of replacing or assisting manual large-scale dialogue data analysis and task definition.