Research On Learning Emotion Analysis And Application Based On Facial Expression Recognition

The rapid development of the Internet has provided a good platform for the popularity of online education,and the current severe situation of the new crown epidemic has further highlighted the importance of distance online education.Compared with traditional offline classes,online education has the advantages of sufficient teachers,rich course selection and not being bound by time and place,but it does not have the advantage of face-to-face emotional communication between teachers and students,and it is difficult for teachers to observe students’ learning emotions,yet students’ learning emotions have a significant impact on teaching effectiveness that cannot be ignored.Therefore,the difficulty of teacher-student interaction is a pressing issue for online education.Students’ facial expressions convey their direct feelings and understanding of the teaching and learning process,and are a direct reflection of their emotions.At the same time,expression recognition,as a non-invasive technology,is highly acceptable to learners and requires low instrumentation.The use of expression recognition technology to analyze students’ learning emotions can help teachers keep track of students’ learning status and respond to undesirable emotions in a timely manner,so that they can design more rationalized teaching content,enhance teacherstudent interaction and improve teaching quality.At present,although considerable progress has been made in expression recognition technology for online education applications,there are still some pressing issues to be resolved.Regarding the dataset,learning emotions have their own specificity,and the corresponding expression categories cannot be generalized according to the basic expression categories recognized in the field of expression recognition,and corresponding expression categories need to be designed for learning emotions.Moreover,face data involves privacy,and there are few open source largescale datasets.This has led to a lack of datasets for learning emotion research in China.Regarding algorithm design,the currently available expression recognition models are usually designed to be complex and large in order to improve recognition accuracy,resulting in high hardware cost requirements for practical application deployment and the need for lightweight research on the models.At the same time,online education has a large number of students attending classes at the same time,and the existing expression recognition models are unable to meet the demand of recognizing the learning emotions of a large number of students at the same time while ensuring the recognition accuracy and inference speed.These problems pose significant challenges for the study of learning emotions based on face expression recognition.To address these problems,this thesis focuses on face expression recognition algorithms in a learning context and develops a learning emotion analysis system.The main work and results achieved in this thesis are as follows:(1)To address the current problem of the vacancy of face expression datasets in learning contexts,a face expression dataset of students’ spontaneous learning emotions in learning contexts is constructed.Firstly,through research on the identification of learning emotions in education,combined with interviews and surveys of students in online classrooms,six categories of expressions reflecting common learning emotions were identified: happy,tired,sad,surprised,bored and neutral.Then,data on students’ spontaneous expressions in real classroom environments were collected and the data were annotated through a combination of model annotation and manual review.Finally,the dataset was expanded by means of web crawlers and data augmentation.The final face dataset LE-FER,which is applicable to the study of expression recognition in learning situations,was obtained,containing 10,000 face images and 6 types of learning expressions.(2)A lightweight face expression recognition model Multi-scale Feature Net(MSFNet)based on multi-scale features combined with attention mechanism was designed to address the requirements of model lightweight and generalization when deployed in practical applications.Firstly,the model is based on the idea of dense connectivity to reuse feature maps,and uses convolution kernels of different scales to obtain multi-scale features and improve recognition accuracy.Secondly,a "progressive" lightweight structure is proposed to achieve decreasing information interaction between channels and to optimize the size of the model while ensuring accuracy as much as possible.Finally,an attention mechanism is introduced into the model to facilitate the efficient propagation of channel-specific information.The model,with only 0.33 M number of parameters,achieves 89.12% accuracy on the LE-FER dataset and performs well on all open source datasets,achieving a lightweight design while ensuring high recognition accuracy and good generalization.(3)A fast and lightweight face expression recognition model,Multi-ghostnet,based on multi-level features combined with dynamic discriminative mechanism is designed to meet the requirement of real-time model recognition in online education system applications.The model is then compressed.The multi-level transformation allows the linear transformations between feature maps to be superimposed sequentially,significantly deepening the network and effectively reducing the performance degradation caused by direct coarse single-level transformations in the network.In addition,an adaptive dynamic discrimination mechanism is proposed to assign weights to the importance of the feature maps at each level to maximize the use of valid information.Finally,a residual join is introduced so as to retain more of the original face information and improve the model accuracy.The model achieves a real-time recognition speed of 25ms/frame while taking into account the recognition accuracy,number of parameters and computational effort,and is suitable for real-time monitoring of learning emotions during large-scale student classes in an online education environment.(4)A learning emotion analysis system is designed and implemented for online education applications.Based on the trained expression recognition model,the Py Qt5 framework was used to build the learning emotion analysis system,and the expression recognition model was deployed to the platform.The system mainly includes several functional modules such as user login,student registration,student sign-in,single person learning sentiment analysis and group analysis.The trial run on site showed that the system was able to recognize one frame every 10 frames,and hundreds of frames were tested continuously,and it was able to achieve real-time and efficient recognition smoothly.At the same time,after Py Installer packaged into.exe format,the system can be flexibly applied to all mainstream operating systems,practicality and scalability is high.By accessing the platform,the teacher side can intuitively access the students’ expressions and learning status,pushing the face expression recognition technology for online education into practical application.