Research On Attention-Based Neural Machine Translation With Encoder-Decoder Architecture

In recent years,with the rapid development of neural machine translation,it has shown a great advantage over traditional statistical machine translation,either from the performance or the complexity of the algorithm.In this paper,the traditional RNN-RNN model with attention-based encoder-decoder architecture is studied.For the problems of insufficient feature extraction,missing information of original word vector of real word,over-translation,under-translation and the scarcity of the data,the improved approaches to alleviate specific problem is proposed start from the three separable sub-modules of encoder,attention mechanism and decoder.As a feature extractor of the attention-based neural machine translation with encoder-decoder architecture,the encoder is equivalent to the cornerstone of the whole architecture.The feature extraction ability of the encoder directly determines the per-formance limit that the translation system can achieve.For the problem of insufficient feature extraction ability of traditional BiRNN encoder,a fusion multi-encoder method is proposed.For the problem of missing information of original word vector of real word caused by the encoder’s no distinction between virtual and real words,the RCNN structure in text classification is introduced to design an enhanced encoder.The experi-mental results show that both the proposed fusion multi-encoder method and enhanced encoder design can effectively improve the feature extraction ability of the encoder and improve the performance of the system.The last few years have witnessed the success of attention-based neural machine translation,and many of variant models have been used to improve the performance.Most of the proposed attention-based neural machine translation models encode the source sentence into a sequence of annotations which are kept fixed for the following steps.In this paper,we conjecture that the use of fixed annotations is the bottleneck in improving the performance of conventional attention-based neural machine translation.To tackle this shortcoming,we propose a novel model for attention-based neural ma-chine translation,which is intended to update the source annotations recursively when generating the target word at each time step.Experimental results show that the pro-posed approach achieves significant performance improvement over multiple test sets.For the problem of the scarcity of the data,this paper focus on how to improve performance using the monolingual corpus.In the use of monolingual corpus,statis-tical machine translation can improve its performance through language model;while neural machine translation is difficult to use the monolingual corpus effectively through this way.In order to solve this problem,this paper proposes a jointly semi-supervised training algorithm,where the candidate translations for non-labeled data are first gener-ated by statistical machine translation and neural machine translation models,and then the candidate translations are selected through sentence-level BLEU.The experimen-tal results have demonstrated the effectiveness of the proposed algorithm.In the NIST Chinese-English translation tasks,the proposed method has obtained an average of 1.35 points BLEU improvement over the baseline system.