Research On Driver Action Recognition Within Intelligent Vehicle

With the continuous development of new energy and information technology,the automobile industry is accelerating the development of new energy,intelligence,interconnection,and digital system.The vehicle plays the role of intelligent terminals more than transportations.Smart cabin pays more attention to the in-vehicle perception,aimed at providing a better and safer driving experience.Driver action recognition is achieved by installing a camera in the car and using computer vision technology to identify the driver action,and then a warning or feedback is given to the driver.Action recognition is challenging because of the computational complexity and difficulty in deployment.Action recognition algorithms based on deep learning usually have better robustness and accuracy,and also have higher requirements on computing resources.However,it’s necessary for the algorithm to run at the terminal devices.It’s impossible to upload the data to the cloud sever for calculation because the delay is too high and the signal will lose when driving in a tunnel.Therefore,the main research of this thesis is the design and the terminal-side deployment of the efficient action recognition algorithm inside vehicle,mainly recognizing gesture actions and driver abnormal behavior such as smoking.This thesis first comprehensively introduces various action recognition algorithms based on deep learning and analyzes their advantages and disadvantages.Then an action recognition network structure of fused 2D convolution and 3D convolution is proposed.To solve the problem of the large computational complexity of 3D convolutional networks,the neural architecture search based on reinforcement learning is used to optimize the 3D convolutional network structure,a method called 3D-NAS based on teacher network is proposed to improve the efficiency.In addition,the terminal-side feedback mechanism is designed and implemented.The proposed model achieves a state-of-the-art result on a public gesture dataset with 95.63% accuracy.On our own dataset,the mobile model also achieves 98.33% accuracy.The design of the neural network structure is the first step to run the neural network model on the terminal-side devices,it is necessary to further optimize for the terminal-side hardware platform and software platform.Different deployment scenarios may have huge gaps in performance between each other.This thesis deploys the neural network model under the Kirin 980 platform.According to the proposed action recognition model with a two-stage network structure,an NPU+CPU terminalside heterogeneous computing deployment method is proposed.Compared with the method using the only CPU,the inference speed is greatly improved.In addition,according to the characteristics of the model structure,the feature sliding window is used to remove the redundant calculation in the scene of the online real-time video stream,and finally,the real-time requirement is achieved.The inference time is only 19.81 ms.