Three-Stream 3D Convolutional Neural Network For Near Infrared Facial Expression Recognition

Facial expression recognition has been widely used in many fields,such as human-computer interaction,biology,security,medical care,and computer manufacturing.In recent years,performance of the research has been significantly improved with the help of deep learning.As far as the current employed face expression data sets are concerned,most of them to a largely extent are gathered under the visible light spectrum,but the difference in illumination brings about much variation in image imaging,and great influence in accuracy.At present,most of the data sets for facial expression recognition have used static single-frame images.However,facial expression changes are involved in dynamic process.Image sequence-based methods are more in line with the essential characteristics of expression generation than methods based on single-frame images.In response to the above problems,we propose near-infrared facial expression recognition method based on a three-channel three-dimensional convolutional neural network.The work of this paper is as follows:(1)3D convolutional neural network(3D CNN)was used for near-infrared facial expression recognition.The network automatically extracts the temporal and spatial features of near-infrared image sequences,effectively utilizing the dynamic information of facial expressions.Although 3D CNN has been proposed before,we have explored near-infrared facial expression recognition in 3D CNN for the first time.(2)A 3D CNN called NIRExpNet containing two sub-networks: global network and local network was designed.The global network extracts global features of the entire face,and the local network extracts local features of parts of the human face(upper and lower of the human face).(3)The NIRExpNet network was designed through experiments.In order to avoid over-fitting phenomenon,a medium-sized 3D CNN was designed through experiments.The global network adopts the VGG-M-2048 network structure,and the local network uses two shallow layers of the convolutional network structure.Moreover,the concatenation fusion method is finally selected to fuse the sub-networks so that the entire NIRExpNet network can better integrate the global and local features and finally reach the optimal state.This article employs Oulu-CASIA near-infrared facial expression database to evaluate the performance.The experimental results illustrate that NIRExpNet achieves 78.42% recognition rate,which surpasses other comparison algorithms(LBP-TOP(72.33%),3D HOG(60.00%),3D CNN DAP(72.12%),DTAGN(66.67%))these proves the effectiveness of the proposed algorithm.