Exploiting Multi-source Data For Zero-shot Cross-domain Dependency Parsing

Domain adaptation focuses on the performance generalization problem outside the independent identically distribution hypothesis,that is,the problem of sharp degradation of model performance caused by potential distribution differences between the test data(target domain)and the training data(source domain).Aiming at domain adaptation dependency parsing,this paper attempts to use data from three sources:unlabeled data in the target domain,conceptual graph data,and labeled data in multiple source domains.Firstly,from the perspective of the target domain,three classic tri-training methods are used to leverage the target domain unlabeled data,that is,the prediction results of multiple models are used to improve the quality of the automatically labeled data.Secondly,from the perspective of the intermediate connection between the target domain and the source domain,this paper introduces a conceptual knowledge base,ConceptNet,to capture the bridge relationship between different domain concepts by using the knowledge-guided domain adaptation framework to enhance the semantic information of sentence encoding.Finally,from the perspective of the source domain,considering that there are often annotated data from multiple source domains in real application scenarios,this paper proposes a feature transformation network for multi-source cross-domain scenarios,which balances the contribution of each source domain by distinguishing between domain public and private knowledge.Specifically,the research content of this paper is as follows:(1)Mining unlabeled data for single-source cross-domain dependency parsing.One of the reasons for the degradation of cross-domain model performance is the lack of highquality target domain labeled data.Since manual labeling data is time-consuming and laborintensive,automatic generation of target domain labeled data is an effective solution.Among them,tri-training,as a typical multi-model decision collaborative training method,aims to use the prediction results of multiple models to ensure the quality of automatically labeled data.In this paper,for the cross-domain dependency parsing,three commonly used tri-training methods are systematically compared,and these three algorithms complete the screening of automatically annotated data from two aspects:data predicted by three models under the same time round,data predicted by two other models other than the current model in the same time round,and data predicted by the model in the previous two rounds of the current round.We have achieved state-of-the-art on the NLPCC-2019 evaluation data and greatly exceeded the current best results.In addition,detailed analysis experiments are designed to deeply understand the reasons for the performance degradation of cross-domain models and the role of tri-training.(2)Infusing graph data for single-source cross-domain dependency parsing.The work on tasks such as dialogue generation,relationship extraction,and language model shows that graph knowledge can further improve the powerful representation ability of neural networks,and the relationships between different nodes in the knowledge graph naturally correspond to the direct or indirect connections between concepts in different domains.This paper uses a knowledge-guided two-stage domain adaptation architecture,which uses the knowledge graph ConceptNet to provide domain-general and domain-specific background conceptual knowledge to improve the semantic information contained in sentence representation.The first stage is to extract the concepts in the original text of all domains and obtain a ConceptNet correlation sub-graph of these concepts,and then use this sub-graph to train a graph convolution network auto-encoder.The second stage is to use the auto-encoder of the previous step to obtain the conceptual representation in the input sentence and input the representation as an additional feature into the classical domain adversarial network.We get a not significant performance improvement on SANCL12 evaluation data,and the analysis experiments showed that the benefit of this method was positively correlated with sentence length,that is,there was a more obvious positive benefit on the longer sentence,and the longer the sentence,the more obvious the improvement.(3)Balancing multiple source domain data for multi-source cross-domain dependency parsing.It is often more practical to have access to data in multiple domains at the same time,and multi-source domain adaptation is designed to adapt to such scenarios.The difficulty of multi-source domain adaptation lies in how to balance the contribution of each source domain and how to make full use of large-scale unlabeled data.Because of the limitations regarding the accuracy of the existing method using unlabeled data and the redundancy of the architecture of the model balancing different domains,a two-stage training algorithm TRI-FTN is proposed for the multi-source domain adaptation dependency parsing.On the one hand,the tri-training method is used to improve the accuracy of using large-scale unlabeled data in the target domain,and on the other hand,the feature transformation network(FTN)is used to balance the contribution of different source domains.The core idea of the feature transformation network is to change the traditional horizontal public-private network into a vertical public-private network and replace the private network with a lighter feature transformation module.The experimental results on the CCL-2021 evaluation data show that compared to vanilla horizontal shared-private network,the performance of FTN is superior in almost every domain with the cost of half of the model parameters and a quarter of the parsing time.Detailed analysis experiments reveal the role of each module of TRI-FTN.In summary,aiming at the zero-shot cross-domain dependency parsing,this paper attempts to use data from different sources to carry out domain adaptation research from the perspective of target domain,the intermediate connection between target domain and source domain,and the perspective of source domain.That is,using unlabeled data from the target domain through different self-learning methods,using knowledge graph data by knowledgeguided two-stage domain adaptation architecture,using domain data from multiple source domains by feature transformation networks.We hope that these contents can bring some inspiration to scholars in the field of domain adaptation.