Research On Key Techniques Of Transfer Learning In Task-oriented Dialogue System

Dialogue systems can provide information or services with a human-computer interface,which has gained increasing attention in both academia and industry.Recently,dialogue systems can be classified into four main categories in terms of functionality:task-oriented dialogue systems,chit-chat dialogue systems,knowledge-grounded question answering systems,and recommendation systems.This thesis focuses on task-oriented dialogue systems that can help users to achieve specific goals(e.g.,checking phone bills,restaurant reservations,and booking tickets),which have high theoretical significance and practical value.In recent years,research on To Ds can be classified into two main categories: pipeline task-oriented dialogue systems and end-to-end task-oriented dialogue systems.Pipeline task-oriented dialogue systems mainly consist of four modules: dialogue language understanding,dialogue state tracking,dialogue policy learning,and natural language generation,which generates system response with the help of four modules together.Meanwhile,end-to-end task-oriented dialogue systems can adopt a unified sequence-tosequence model to directly generate system responses.Dialogue language understanding,the first module in the pipeline task-oriented dialogue system,is used for extracting the user’s semantic result for subsequent modules to produce an appropriate system response,which is one of the core modules in the pipeline task-oriented dialogue system.With the development of deep learning and pre-trained models,great progress has been made in both pipeline and end-to-end task-oriented dialogue systems.However,in real-world scenarios,they still face a major challenge: data scarcity problem.Current models heavily rely on large amounts of high-quality annotated data for training,resulting in poor generalization in a low-resource setting.Meanwhile,unlike other natural language processing tasks,data annotation in the To Ds domain often requires expert knowledge,making it more difficult to annotate data.In addition,when scaling to different tasks,different domains,and different languages,it is inevitable to suffer from data scarcity problems.As a result,how to build a task-oriented dialogue system with considerable performance in a data scarcity setting is a very meaningful research topic.Therefore,this thesis explores the application of transfer learning to To Ds: aiming to transfer knowledge from the source domain with sufficient data to the target domain with limited data by transfer learning to alleviate the data scarcity problem in the taskoriented dialogue system.Specifically,this thesis explores cross-task transfer,crossdomain transfer,and cross-lingual transfer on dialogue language understanding in the pipeline task-oriented dialogue system and the end-to-end task-oriented dialogue system.·(Cross-task Transfer)A Stack-Propagation Framework for Cross-task Dialogue Language Understanding: The current dialogue language understanding model can only make an implicit joint interaction between intent detection and slot filling,resulting in insufficient intent information transfer.This thesis proposes a stack-propagation framework to explicitly incorporate the intent detection information.In addition,to alleviate the problem of error propagation caused by the introduction of intent information,this thesis further proposes a token-level intent detection mechanism,which provides tokenlevel intention information for each word.This work not only achieves state-of-the-art performance on two low-resource benchmarks but also improves the interpretability of the dialogue language understanding framework.In addition,with extremely limited data(5% training data),the proposed stack-propagation framework gains 19.7% performance over the previous work,which greatly alleviate the data scarcity problem.·(Cross-task Transfer)A Co-Interactive Transformer for Cross-task Dialogue Language Understanding: Though previous work has explored the interaction between intention detection and slot-filling tasks,they are still limited to considering the knowledge transfer from intent information to slot filling while ignoring the knowledge transfer from slot filling to intent detection.To address this issue,this thesis proposes a cointeractive transformer for joint slot filling and intent detection.Different from the vanilla Transformer,this thesis introduces a co-interactive attention module to model the relation and interaction between the two tasks,aiming to consider the cross-impact of the two tasks.In addition,this thesis further extends the feed-forward network layer to better implicitly fuse intent and slot information.This work outperforms the best previous single flow information interaction framework by 26.7% in data scarcity data setting(5% training data),which can alleviate the cold-start problem of dialogue language understanding in the real-world deployment.·(Cross-lingual Transfer)A Multi-lingual Code-switching and Contrastive Learning Framework for Cross-lingual Dialogue Language Understanding: To improve the generalization of the system to cross-lingual scenarios,this thesis first proposes a multi-lingual code-switching framework to implicitly align the representation of multilingual pre-trained models across languages.In addition,the thesis further introduces a contrastive learning method to explicitly align representation across languages.Specifically,this work employs contrastive learning to encourage the representations of positive samples to be more similar than negative samples,which achieves explicitly align representations of similar sentences across languages.This framework achieves state-of-the-art performance on the zero-resource cross-lingual dialogue language understanding task,greatly advancing to generalize real applications to different national languages.·(Cross-Domain Transfer)Dynamic Fusion Network for Multi-Domain Endto-end Task-Oriented Dialog.Current dialogue models rely on large training data that are only available for a certain number of task domains,making it hard to generalize other low-resource domains.To address this problem,this thesis proposes a dynamic fusion network(DF-Net)to automatically learn the relevance of different domains.Results show that our framework consistently and significantly outperforms the current state-of-the-art methods.In addition,with limited training data(5%),DF-Net outperforms the previous best methods by 13.9% on average.which greatly improves the generalization of the model to cross-domain scenarios.In summary,this thesis aims to address the problem of data scarcity problem in task-oriented dialogue systems.Specifically,this thesis systematically investigates crosstask transfer,cross-domain transfer and cross-lingual transfer on both dialogue language understanding in pipeline task-oriented dialogue systems and end-to-end task-oriented dialogue systems,which significantly alleviate the data scarcity problem and improve the generalization of the model in the cross-task,cross-lingual and cross-domain scenarios.