Research On Deep Learning Based Methods For Driver Action And Drowsiness Recognition

Along with the development of economy and the advancement of urban modernization,vehicles have gradually entered thousands of families and become an indispensable tool for transportation.Due to the increase in vehicle population,the traffic accident rate sharply rises and road safety has become a hot topic of social concern.According to the survey,more than 80% traffic accidents are closely related to unsafe driving.Therefore,it is very necessary to employ intelligent device for real-time monitoring,as well as analyzing driver’s action and state automatically.In this paper,driver in surveillance video is taken as the research object;it focuses on two key issues,including driver action recognition and driver drowsiness recognition,respectively.Aiming at the problem of driver action recognition,this paper proposes a multi-scale convolutional network-based approach,which is suitable in the application with embedded platform.Besides,a spatial-temporal convolutional network-based approach is proposed for driver action recognition,which can be applied to general computer platform with better recognition accuracy.These driver action recognition approaches are robust to the illumination variations in both daytime visible lighting scene and nighttime infrared lighting scene.Compared with driver’s action feature,driver’s facial representation is more susceptible to the illumination variations.Aiming at recognizing driver drowsiness under complex illumination condition,this paper proposes a generative adversarial network based approach for face illumination processing,which realizes facial shadow removal and detailed information recovery.From this foundation,this paper proposes a two-frequency spatial-temporal convolutional neural network-based approach for driver drowsiness recognition,which fulfills the application requirement in various complex illumination scenes during daytime and nighttime.The main research contents and innovations are summarized as follow:(1)The multi-scale convolutional network-based approach is proposed for driver action recognition.Driver action recognition belongs to the category of fine-grained classification in the pattern recognition domain.Existing driver action recognition approaches remain deficiencies in fine-grained feature representation;moreover,these approaches fail to reconcile the demand between recognition accuracy and computational efficiency,which are not applicable in embedde platform.Based on it,the multi-scale convolutional neural network is designed in this paper;The proposed model not only applies lightweight technology to the multi-scale convolution operation,but also realizes adaptive feature-guided learning through visual attention mechanism.Experimental results demonstrate that the proposed driver action recognition approach not only ensures high accuracy,but also decreases model parameters and reduces computational complexity,which is suitable in embedde platform.(2)The spatial-temporal convolutional network-based approach is proposed for driver action recognition.Under satisfying the premise of high hardware configuration,spatial-temporal convolution is more suitable than spatial convolution for modeling the problem of action recognition.Existing spatial-temporal feature extraction models are difficult to perceive subtle and slow motion cues effectively;moreover,these models do not make the full use of the spatial and temporal saliency information,that is,there are still some limitations in actual driving environment.For resolving this problem,a hybrid spatial-temporal convolutional network is designed in this paper.The proposed model integrates convolution neural network and recursive neural network,which realizes long-range spatial-temporal feature fusion with two-stage network training;moreover,it implements the refinements of driver action cues through multi-task learning strategy and visual attention mechanism.Compared with the multi-scale convolutional neural network,the proposed spatial-temporal convolutional neural network significantly improves the accuracy,as well as increses the computational complexity,which is suitable in general computer platform.(3)The generative adversarial network-based approach is proposed for driver’s face illumination processing.Complex illumination is one of the key factors that affects facial feature extraction,while illumination processing is a basis for recognizing facial drowsiness state.Existing face illumination processing models do not make the full use of intrinsic information;moreover,these models are difficult to enhance the illumination component adaptively.Based on this,a novel face illumination processing model is proposed in this paper.Specifically,the face illumination processing task is divided into two stages: image decomposition stage and illumination enhancement stage with the combination of self-supervised learning algorithm and generative adversarial learning algorithm.Consequently,the proposed model is guided to generate illumination normalized face images through prior knowledge and conditional constraints.Experimental results show that the proposed face illumination processing approach effectively eliminates illumination variations and completely restores facial details,which builds the foundation for the research of driver drowsiness recognition.(4)The two-frequency spatial-temporal convolutional network-based approach is proposed for driver drowsiness recognition.Facial feature extraction is essential for driver drowsiness recognition.Most of existing diver drowsiness recognition approaches do not consider the influence of complex illumination;moreover,these approaches individually extract global spatial-temporal cues or facial detail cues,which lack the overall representation of the drowsiness state.Based on this,a novel driver drowsiness recognition algorithm is proposed in this paper.In terms of network structure,a two-frequency spatial-temporal convolution network model is designed,where the high-frequency spatial-temporal network branch extracts the global spatial-temporal cues and low-frequency spatial network branch captures the information of facial details;moreover,the lateral connection mechanism is introduced for feature fusion;In terms of data processing,the proposed model treats the original face image sequence and the illumination-normalized face image sequence as two input data streams;it realizes illumination-invariant drowsiness feature extraction through fusion.Experimental results show that the proposed approach effectively improves the accuracy of driver drowsiness recognition with high detection rate and low false alarm rate,which meets the application requirements of complex lighting environment.