Two Stream Neural Networks For Action Recognition In Radar

In the field of computer vision,more and more people pay attention to the recognition of human action,which become one of the hot spots in engineering and academia in recent years.At the present stage,most of the tasks of human action recognition usually use the information obtained by optical devices such as cameras as the input source and carry out researches on this basis.However,these optical devices are usually subject to the factors of natural environment,such as illumination,temperature,and also to factors such as angle of view,which bring some trouble to the recognition tasks.Radar signals can be used to avoid these interference factors,so the radar signals gradually enter the researchers’ field of vision.At present,most of the research is based on micro-doppler images and combined with deep learning to recognize human action and made some progress.However,it only makes use of the frequency domain information,ignores the information in other domains.In this regard,we used two-stream neural network to deal with this task in this paper.The main contribution of this paper is follow:(1)Two-stream neural network is proposed and applied to the radar domain for human action recognition.This method combines the information contained in spatial domain with the information in the frequency domain,and simultaneously uses the two kinds of information to carry out the human action recognition task.In two-stream neural network,two independent convolutional neural networks are used to extracted the features in the frequency domain and the characteristic information in the spatial domain,and then the features are combined and classified by a fully-connected neural network.By the network,we realized the purpose of combining the spatial domain information and the frequency domain information.The results show that accuracy of the two-stream convolutional neural network is about 4%higher than the single-stream convolutional neural network using spatial domain information or frequency domain information.(2)Two stream neural network based on feature fusion of LSTM is proposed and applied to human motion recognition in radar field.In the above network,the convolutional network characteristic of oneself cannot make use of information on the time domain,and there are multiple connection layer with larger computational overhead problem,therefore,the above network structure is improved and a new method of human action recognition based on long short-term memory(LSTM)feature fusion is proposed.Compared with the network mentioned above,the network will use LSTM for feature fusion,so as to introduce the characteristics in the time domain for the recognition,and it can reduce the parameters.Using the same data set for comparison,it can be found that the new network structure improved the accuracy from 90.67%to 93.99%.(3)Two-stream neural network based on early fusion and LSTM fusion is proposed and applied to human action recognition tasks in the field of radar.This paper will improve the feature extraction part of the network proposed above so that more feature information can be introduced.At the shallow level of feature extraction,the information features in two different domains were fused in advance to form a new mixed feature stream,which introduced more feature maps.All the new feature maps and the feature maps of two different domains are used to further extract and then merge the features for classification,so as to achieve better performance.Compared with the above experiment,the accuracy of the new network is 3%higher than the network above.