Research On Recurrent Neural Network Model With Internal And External Memory

Artificial neural networks have been demonstrated outstanding performance in many practical fields,such as natural language processing,speech signal processing and image processing.As a type of the artificial neural networks,recurrent neural networks are remarkably adept at processing time series tasks,which is mainly because of its ability to remember timing signal.As one of the most important mechanisms in the recurrent neural network model,memory mechanism plays a significant role in improving the network learning ability.With a good memory mechanism,the network can combine the input signal with the long-term memory signal to participate in the decision-making of the output,thereby improving the learning ability of the network.However,traditional recurrent neural networks can not keep the memory information for a long time.On the one hand,the learning conflict problem(i.e.input weight conflict,output weight conflict,memory conflict)and the gradient vanishing problem make the network difficult to learn the memory signals effectively.On the other hand,the memory capability problem makes the network has a very limited memory capability to store the memory information.In order to solve the above problems,this paper proposes two types of recurrent neural network model,which are: 1.Recurrent neural network with internal memory units and external memory matrix(RNN-IEM).2.Recurrent neural network with gated memory units and external memory matrix(RNN-GEM).Generally,the main contribution of this paper can be summarized as follows:1.In order to conquer the memory conflict problem,the gradient vanishing problem,and the memory capability problem,this paper proposes a recurrent neural network model with internal memory unit and external memory matrix.The internal memory unit is composed of a linear neuron,which can be used to store the memory signals individually thereby solve the memory conflict problem.In addition,the linear design of the memory unit allows the gradient signals to propagate to the previous time constantly,which can be used to solve the gradient vanishing problem.The external memory is used to store the memory information structurally,which eliminates the memory capability problem of the recurrent neural network.Furthermore,this paper combines the truncation technique to develop an efficient learning algorithm for the proposed model,which makes the model can be trained efficiency.2.In order to conquer the input conflict problem and the output conflict problem,this paper proposes a recurrent neural network model with gated memory unit and external memory matrix(RNN-GEM),which is based on the proposed RNN-IEM.The RNN-GEM has a similar structure with the RNN-IEM,the main difference lies in the structure of the hidden layer,the hidden layer of RNN-GEM is composed of the gated memory unit,the introduction of the gated memory unit makes the decision of the input and output information no longer determined by the weight signal individually,and also consider the influence of the gate memory unit,which effectively solves the input and output weight conflict problem.In addition,in order to make the model training efficiency,this paper also proposes an efficient learning algorithm for the RNN-GEM,which further enhances the learning ability of the network.At last,this paper conducts the experiment on three different tasks,which are the Embedded Reber Grammar(ERG)sequence generation task,the Synthetic World Model(SWM)question answering task and the Language Understanding(LU)entity recognition task.The results show that the proposed model combining with the designed learning algorithm obtain the outstanding performance in the efficient training process.