Research And Application Of Recurrent Neural Network Performance Optimization Method

In recent years,research in neural networks and deep learning has achieved impressive results along with the wave of the information age,with large-scale research taking place in various fields.Recurrent neural networks are a common neural network structure,and are widely used in various natural language processing tasks due to their ability to process sequential data.However,recurrent neural networks alone are not able to handle sequence data of indeterminate length,so researchers have further proposed sequence-tosequence models based on recurrent neural networks,and these methods and techniques have been the first choice of researchers when dealing with sequence data due to their good results.However,because of the increasing size of neural networks,their performance problems have been the shackles that limit their better performance.This thesis combines the optimization of recurrent neural network models with the optimization of dynamic neural networks,which are optimization methods that give neural network models the ability to adjust the network structure or parameters according to different input samples during operation.Such optimized models have greater advantages in terms of efficiency and representational power than traditional static neural network models.In this thesis,two main optimization methods are used for recurrent neural networks(simple recurrent networks,GRU,LSTM).One is the optimization method using locality sensitivity Hashing for the fully connected layer in the recurrent neural network.In this thesis,based on the technique of training neural network models with locally sensitive hashing,the locality sensitivity Hashing is applied to the inference process of the fully connected layer of the recurrent neural network model.The method uses the fast nearestneighbour retrieval property of locality sensitivity Hashing to select different neuron units to be activated for each input sample of the fully connected layer,thereby compressing the matrix size of the fully connected layer and achieving a speedup in the operation of the fully connected layer.In this thesis,by adjusting the usage and parameter settings of the optimization method,we achieve a runtime saving of up to 20%for the original model.The second is a predictor-based skip optimization method,which trains the predictor so that during the operation of the recurrent neural network,the predictor decides whether to skip the computation at the current moment based on the input at the current moment and the information at the previous moment,thus improving the overall running speed of the model.In this thesis,the original method is applied to a more complex recurrent neural network model,and adjustments are made to the use of the optimization method based on the actual model,resulting in a running time saving of up to 18%for the original model.Based on the above optimization methods,this thesis further compares and applies the two optimization methods to a sequence-to-sequence model to achieve an optimization of the sequence-to-sequence model,resulting in a runtime saving of up to 9%for the original model.