Expression Recognition Algorithm And Its Application In E-learning Environment

Human behavior and cognition will all be driven by emotion.Therefore,emotion recognition is a key way to look insight human evolution.Facial expression recognition is an exceedingly essential part of emotion recognition,because in daily communication,55%of the information is conveyed through expressions.Facial expression recognition refers to computers' intelligence of classifying human emotions from facial images.It is a hot research subject in visual technology,and makes a valuable contribution towards the development of various disciplines.In the file of education,facial expression recognition also exerts a growing important effect.Modern education increasingly emphasizes the development concept of "student-oriented",so the measurement of students' cognitive load is very important.Cognitive load reflects the students' interaction ability between psychology and information,which can provide crucial guidance for teaching and learning.Overload is a common plight,especially in e-learning scenes,where learning resources are rich and the amount of knowledge is abundant.Therefore,identifying students' emotions through facial expressions applying the emotions to cognitive load measurement will conduce to individualized teaching,which is enormously beneficial for promoting students' learning and healthy growth.With the progresses of related technologies such as deep learning,expression recognition has shown rapid development.However,the performance of current algorithms is still far from meeting actual application requirements.Especially in natural scenes,such as e-learning environment,due to the influences of head pose changes,the correlations/ambiguities of expressions and the complexities of images,facial expression recognition is still a major challenge in the field of visual technology.The main problems are:(1)the self-occlusion influences from head pose changes.Self-occlusions will lead to huge losses of key information,and cause large changes in the facial appearances,bringing difficulties to facial expression recognition;(2)over-fitting in deep expression recognition.Expressions in natural scenes are routinely multiple,and thus are of great correlations/ambiguities.Hard labels can't describe real expressions well,and usually lead to over-fitting problem,which will make well-trained algorithms collapse in practical application;(3)the difficulty of extracting discriminative features.Expression images under natural scenes are generally with high complexities.As significant changes exist in background and illumination intensity and faces are mixed with many irrelevant attributes,images always couple with high noise.In the teaching environment,the students'expressions are not significant.In addition images captured in such scene usually have low resolutions.Therefore,when applied in e-learning environment,it is more difficult to extract discriminative features,which will easily lead to expression recognition performance degradation.In view of the above challenges,basing on deep learning,this dissertation proposes novel algorithms to improve the performance of expression recognition,and further develops application research for the proposed algorithms in cognitive load detection.Aiming at the above goals,the main research contents of this dissertation are as follows.(1)Developing a head pose-robust deep smile detection model,to solve the self-occlusion problem.Smile is the most common expression of human beings,and smile detection has a wide range of application prospects.This research proposes a robust smile detection framework based on convolutional neural networks,which can alleviate the influence of head pose changes and thus enhance detection performance.The proposed framework customizes two layers:a)constructing a smile feature extraction layer using hidden factor analysis.This research introduces hidden factor analysis into the deep learning network,helping to learn features against self-occlusions;b)constructing a discriminative feature extraction layer using margin fisher analysis.Margin fisher analysis is a data mining model,which can map data into space where classes are easier to separate.In this dissertation,margin fisher analysis is introduced into deep learning network to further learn the features with high discrimination,rewarding to enhance smile detection accuracy of the network.These two layers serve as fully connected layers,and are connected layer by layer to the backbone of the deep network.The comparisons with advanced methods show that the proposed one can achieve better smile detection performance.(2)Developing a deep expression recognition model boosted by soft label and diverse ensemble,to solve the over-fitting problem.The proposed model includes three modules:soft label construction,deep regularization training of base classifier,and ensemble prediction.Specifically,a)the soft label constructor constructs probability distributions of expressions--label vectors with confidence level less than 100%,which can describe the true expression classes and the correlations/ambiguities with other classes,and provide a global relational views of expressions.The soft label constructor contains two main steps.In the first step,constructor model is trained using hard label as supervision.In the second step,the soft labels are obtained by fusing the potential probability vectors from predictions;b)base classifiers are trained using the soft labels,which provide more logically related information as supervision,promoting the regularization training in deep learning and thus can effectively prevent over-fitting.In addition,a label perturbation strategy is proposed,which changes the regularization effect by perturbing the soft labels and thus ameliorates the diversity of base classifiers;c)in the prediction stage,score average is used to make an ensemble for base classifiers to get a strong classifier.The experiments on public databases demonstrate that the proposed method can effectively prevent over-fitting.(3)Developing a multiple attention network for facial expression recognition,to solve the problem of discriminative feature learning.In this research,visual attention from coarse to fine is introduced into deep learning network,and attention under different receptive fields are connected to achieve enhanced attention.Attention mechanism can effectively intensify key information while suppressing irrelevant one,and can therefore improve the learning of discriminative features.In the proposed multiple attention network,a region-aware subnet learns location masks of expression related regions from coarse to fine,and the expression recognition subnet uses the mask to fuse multiple visual attention.In the expression recognition subnet,the multi-attention module contains a hybrid attention branch with multiple sub-branches,and each sub-branch learns the attention for a current region,decoupling the noise features.The multi-attention module also contains a weight learning branch,which is used to learn the importance of different regions adaptively,and to integrate the diverse attention.Compared with the similar methods,the proposed one has better practical application potential.(4)Developing a cognitive load detection model for e-learning using facial expression recognition.The existing researches of cognitive load lack of emotional measurement,and thus have some limitations.This research aims at e-learning environment,and the emotion of students in the learning is measured by the proposed expression recognition algorithm.In addition,task performance,subjective evaluation and other indicators are also measured.Multi-dimensional indicators are integrated for comprehensive detection of cognitive load.In this research,the detection results of students' cognitive load are compared with their real levels,which shows the validity of the proposed method.In summary,this dissertation proposes novel algorithms based on deep learning.Related researches effectively boost the performance of facial expression recognition in nature scene,which will accelerate the development of this field.In addition,this dissertation conducts research on the application of proposed algorithm in cognitive load detection in e-learning,which can provide theoretical and technical support for the optimization of teaching.And this research would benefit the advance of interdiscipline upon artificial intelligence,computer vision,and intelligent education.