Optimization Of Recurrent Attention Model Training

Deep learning has achieved great successes in many fields such as computer vision,machine translation and voice recognition.Application of deep neural networks is advancing the state of the art on several data sets.These exciting results,however,come from a high computational cost at training and testing stage.In computer vision,many methods have been raised for improving the computational efficiency(e.g.reducing the sliding widows number).However,the cost still scales linearly with the size of image.To solve this problem,we introduce attention mechanism,which tries to simulated the vision process of human being.One of bottlenecks of traditional deep models is that they need to deal with the whole input images while human eyes just focus on different interesting parts of image at different times.This vision feature,also called attention,saves “bandwidth” of human vision input significantly.There are two major types of computer attention models: soft attention and hard attention.Soft attention models are based on a differentiable saliency map;while hard attention use discrete locations to attend glimpse areas and generate attention features.Based on recurrent attention models(RAM),one of hard attention models,we proposed two optimization strategies: OV-RAM and EM algorithm.These two methods are then tested on weakly-labeled data sets Translated MNIST and Cluttered MNIST.Recurrent attention model,based on RNN,is capable to attend different glimpse locations and update its internal state at every time step.And finally it makes decision based on accumulated glimpse information.Because it deals with a limited input as a part of the whole image,recurrent attention model can significantly reduce the cost of both training of testing.However,as a result of intractable posterior expectations and high variance for the stochastic gradient estimates,training such an attention model is very difficult.Borrowing techniques from previous studies,by combining soft attention and hard attention,we introduced a new attention model called OV-RAM,which is composed by adding an extra overview layer to RAM to provide context information.And also,we targeted the main issues in RAM architecture.Given the derived formula of a new learning rule with form of two components coupled in RAM architecture,we applied our EM algorithm on it to train RAM.At last,we analyzed some failed examples and then brought forward some available tricks to fix it.We trained and tested our methods on translated MNIST and cluttered MNIST.Experiment results has showed that our methods,OV-RAM and EM algorithm,can efficiently promote the learning rate of attention model.Our methods are proved to be effective by achieving same coverage accuracy with less training steps than the original RAM.