Research On Driver Fatigue State Recognition Method Based On Face And Heart Rate Feature Fusion

In order to solve the problem that it is difficult to detect the early fatigue characteristics of the driver’s face,which leads to the failure of the driver’s driving assistance system to recognize the driver’s fatigue state in time,this paper studies the principle and method of driver’s fatigue state recognition based on the fusion of the facial and heart rate characteristics by comprehensively using the optical information of the driver’s face state and the photoelectric information of the driver’s heart rate during driving.It is of great significance to improve the level of real-time intelligent recognition of driver fatigue state and promote the development of advanced vehicle driving assistance system.The research work was supported by the 2016 major project of industry university research collaborative innovation in Guangzhou(special project of industry university research collaborative innovation alliance)(201604046006).Firstly,the paper discusses the methods of driver fatigue state recognition and the application of related products based on subjective evaluation,facial features,physiological features,operation behavior and vehicle state features,and summarizes some deficiencies and some development trends;then,on the basis of the discussion of the definition,influencing factors,characterization indexes and research methods of driving fatigue,it carries out In this paper,the driver fatigue state simulation experiment design is put forward,and the driver fatigue state simulation experiment scheme which integrates driver self-evaluation and expert evaluation is put forward.The driver fatigue state experiment platform mainly consists of low intrusive camera and smart watch is constructed,and the driver fatigue state simulation experiment is carried out,and the driver’s face image information under different driving fatigue state is obtained Photoelectric information of rest and heart rate and subjective evaluation grade information of driving fatigue state.Then,a method of driver’s multi facial feature extraction based on OPENFACE is proposed.Five kinds of 51 dimensional facial features,including driver’s head posture,line of sight direction,facial micro expression,eye opening and mouth opening,are obtained from the driver’s facial image information under different driving fatigue conditions.A method of driver’s multi heart rate feature extraction based on smart watch is proposed,By using this method,7-d heart rate and heart rate variability characteristics of drivers,such as instantaneous heart rate,mean heart rate,SDNN,RMSSD,LFP,HFP,LFP / HFP,are obtained from the photoelectric information of drivers’ heart rate under different driving fatigue conditions.Then,the correlation between multi face and multi heart rate characteristics of drivers and their subjective evaluation level of driving fatigue is analyzed,and the multi face of drivers is studied And the statistical difference of multi heart rate characteristics and the significance of the difference,as well as the information fusion relationship between multi face characteristics and multi heart rate characteristics of drivers.Next,the principle and method of driver fatigue state recognition based on the fusion of facial and heart rate features are proposed,the fusion information features of driver’s multi facial and multi heart rate based on feature layer information fusion are analyzed,and the driver fatigue state recognition model based on LSTM network with the fusion information features as input is established.Finally,the data set of face and heart rate fusion features for experimental verification is constructed,and the driver fatigue state recognition model is trained and tested with its training set and test set,and the model recognition results are compared and analyzed from the two aspects of intra method comparison and inter method comparison.The experimental results show that the recognition accuracy of driver fatigue state based on the fusion of face and heart rate features can reach 91.0%,81.9% and 87.5% for "awake moderate severe" driver fatigue state,and the average recognition accuracy can reach 86.8% The recognition accuracy of "sober" and "moderate fatigue" can be improved by 4.3% and 6.7% respectively compared with the method based on facial features alone,which can achieve more accurate recognition of "sober moderate severe" driver fatigue state.