Driver Fatigue Detection Based On Deep Learning And Multi-source Facial Dynamic Behavior Fusion

Driving fatigue is one of the most important causes of traffic accidents.Research on fatigue detection and warning technics is of great significance to improve the safety of drivers and pedestrians.Machine vision technology has been proved to be the most promising technology in detecting driver fatigue.However,there are still many challenges posed by complex illumination,the effectiveness of fatigue characteristics,and driver’s individual difference when a driver becomes fatigue.Based on the above problems,this paper focus on the key issues in driver’s pupil position detection method,eye state recognition method,and driver fatigue expression recognition method to analysis the driver facial dynamic behavior.On this foundation,a driver fatigue recognition model based on adaptive parameters is built.The main work is as follows:(1)Driver’s pupil position detectionConsidering the complex driving environment,a pupil and eye corner detection method based on supervised descent method,weighted traversal integral projection and local binary feature is proposed to locate the pupil position from coarse to fine.First,a face alignment algorithm based on supervised descent algorithm is used to locate and track facial landmarks,and eye corner coordinates are extracted as the initial positions.Then,a weighted traversal integral projection is used to roughly locate the pupil center.Finally,according to the initial points of pupil and eye corners,the local binary feature value of landmarks is extracted by training random forest model,and the precise positions of the pupil and the corner of eye are obtained by the global cascade regression algorithm.(2)Driver’s eye state recognitionA model based on a deep integrated neural network is proposed to classify driver’s eye state.This model combines deep neural network and deep convolutional neural network to construct a fusion model,and then a joint cost function is built according to the output of the two models.The joint cost function is opimized to obtain the optimal model to recognize the closeness of driver’s eyes.To improve the accuracy and robustness of the proposed model,two datasets,which have been used in different fields,are adopted to pretrain the model to initialize parameters.The pretrained model is then finetuned using the target dataset to obtain the final eye state recognition model.(3)Driver facial dynamic fatigue expression recognitionThree driver dynamic fatigue expression recognition models based on image texture and deep learning model are proposed and compared respectively.The main contents are as follows:A dynamic fatigue expression recognition model based on temporal and spatial local binary pattern(LBP)and Gabor fusion model is proposed.First,the dynamic texture features in x and y direction of driver’s local facial region are extracted by using temporal and spatial LBP,and the static texture features are extracted by using Gabor fusion histogram.Then,the two features are fused as fatigue feature parameters.Finally,a support vector machine classifier is used to classify the extracted features to obtain the final results of fatigue expression.A dynamic fatigue expression recognition model based on multimodal learning is also proposed.First,the algorithm extracts local gray image of eye and the coordinates of facial landmarks as the multi-source input features.Then,a multimodal deep learning model based on stack denoising autoencoder is adopted to learn the two inputs for feature extraction.The two features are fused in the abstract layer to obtain the fusion information of facial dynamic behavior.Finally,a softmax classifier is added to the top layer to classify the dynamic fatigue expression.Moreover,a dynamic fatigue expression recognition model based on local ensemble convolutional neural network(LECNN)and encoding vector is proposed.First,driver’s eye and the mouth images of each frame are extracted as input data according to the facial landmarks.An LECNN model that consists of three convolutional neural networks is used to recognize driver fatigue expressions in the static images.Then,the output of the LECNN of each frame in the video is cascaded to construct an encoding vector.Finally,k nearest neighbor classifier is used to classify encoded vector to obtain the driver’s fatigue expression state in the video.According to the experimental results,the recognition accuracy of the three models for dynamic facial fatigue expression recognition are compared and analyzed.The optimal recognition model is obtained according to the recognition rate.(4)Building driver fatigue detection modelAccording to the characteristics of facial dynamic behavior during driver fatigue,an adaptive driver fatigue recognition model,which combines general and individual differences features,is proposed.First,different aspects of facial fatigue features,including pupil motion parameters,eye state parameters,and fatigue expression parameters,are extracted.Analysis of feature difference saliency is conducted to study whether there is a significant difference in each parameter in different fatigue level.According to the analysis results,9 facial behavior features are used to build general feature space.Then,according to the consubstantiality regularity and individual difference of driver’s facial behaviors in different fatigue level,an individual differences feature space is built based on the changes of general features.The two feature spaces are combined to build a more complete adaptive fatigue feature space.Finally,an adaptive driver fatigue detection model is proposed based on the general and adaptive feature space to detect driver fatigue.