Attentional Neural Machine Translation With Syntactic Knowledge Integration

With the rapid development of deep learning,the performance of machine translation models based on neural networks has been rapidly improved and surpassed the traditional statistical machine translation gradually.The neural machine translation model mainly models the entire translation process through the framework of "encoder-attention-decoder",where the attention mechanism is mainly used to model the correspondence between source-side information and target-side information.Simultaneously,syntactic knowledge is also very important for translation,which can help model the syntax representation of word sequences and reduce syntax errors in translation.Therefore,improving the attention mechanism and integrating syntax prior knowledge has great significance for improving the performance of machine translation models and the quality of translation results.For two key research points of attention mechanism and syntax knowledge,we proposes two pieces of research works as follows:1.Neural machine translation with attention based on syntactic branch distance: In the traditional attention mechanism,the linear distance between words determines the translation correspondence between the source and target words.This attention directed by linear distance only considers the linear order of words,ignoring the syntactic structure of sentences and the dependency relations between words,which affects the translation quality of the system.We propose a syntactic branch distance based on the dependency tree corresponding to sentence,captures the syntactic branch distance constraint for each word in the sequence,and direct attention based on syntactic branch distance to focus the source words that are linearly related and syntactically related to the predicted target word,which can learn the long-distance dependence between words,and calculates a more effective context with syntactic constraints to predict the target word.This method adopts the syntactical structure without giving up the attention to the sequential structure,reduces the noise brought by the syntax tree itself to some extent.In the IWSLT English to German translation task,the experimental results show that our model’s translation results are better than other baseline attention system,which proves the effectiveness of our model.2.Syntax-Aware Attentional Neural Machine Translation Directed by Syntactic Dependency Degree: Neural machine translation not only has computational overhead and fails to learn potential linguistic information in attention,but also has the inaccurate understanding for the sentences and the imprecise expression of context,and there are errors in grammar,morphology,and word order disorder,which caused by neural machine translation without integrated syntax knowledge.Existing method based on linear neural networks combined with grammar usually difficult to balance training efficiency and sufficient syntactic information at the same time.Most of the use of dependency grammar is limited to a qualitative perspective,while the traditional syntax distance quantifies the dependency relation on the same extent of roughness,and it is easy to appear that the intimacy of the dependence relationship does not match the syntax distance.Considering that the tree neural network can directly learn the dependency tree,we designs a variant of the tree neural network with a dependency capture layer to capture the syntactic dependency degree of the connected words on the tree,and proposes two distance mechanisms based on the syntactic dependency degree,finally use syntax-aware attention directed by two distance mechanisms to generate source-dependent context,then predict target words.On the IWSLT English to German translation task,experiments show that this method significantly improves the translation quality,and proves that the syntactic dependency degree increases the attention from supplement of dependency syntaxtic information,there by obtaining more effective and abundant context vectors,express the context more accurately.Other experimental analysis proves that our method has undergone careful parameter selection and analysis.This thesis aims to improve neural machine translation based on attention by taking syntax knowledge into account.Both two methods can achieve desirable performance improvements.