Research And Implementation Of Adaptive Tutoring System Based On Multi-modal Emotion Recognition

Online education has become a way for more and more people to get education as it breaks through the limitations of time,space.It also has rich educational resources and low barriers.However,online education is a oneway teaching process due to the lack of interaction between the learner and the teacher interaction.Real-time detection of the learners’ mental states and personalized assistance can effectively alleviate the “emotional loss” problem.Therefore,the research of adaptive teaching system based on multi-modal emotion recognition has important research significance and application guidance value.There are still some problems in the research of emotion recognition in the field of online learning at present.First,video watching is the primary responsibility for imparting knowledge which has received little attention.Secondly,as learners watch videos,learner’s facial portrait data and video interactive behavior data can be obtained through computer technology.However,few studies currently consider the real-time interaction of learners in the emotion model,and most of them do not consider combining the video interaction data and portrait data.Thirdly,current research only stays at the theoretical level and doesn’t reflect the practical application value.In view of the above problems,this paper takes the learner’s video recognition as the main line of emotion recognition and puts forward an adaptive teaching system based on multi-modal emotion recognition.The main results of this paper are summarized as follows:(1)In view of the video learning process,this paper proposes to use the multi-modal model,i.e.,the video interactive behavior feature and the learner’s portrait feature,as the feature data for real-time emotion recognition.The portrait data is acquired and extracted by the camera and the corresponding characteristic data is extracted,and the interactive characteristic data of the video is defined and extracted by the education system.In order to solve the problem of illegal data and imbalance in the data,we firstly pre-clean the data and then use the Borderline-SMOTE1 oversampling method and the Tomek Link Removal undersampling method to generate a balanced data set.(2)Real-time emotion model.This study summarizes past emotional models and proposes a multi-modal emotional model,which collects data using a self-designed client.The multi-class emotion classifier based on multi-modal training is obtained through machine learning algorithms.The learning emotions that this study focuses on are: delighted,confused,concentrated,distracted,surprised,thinking,normal and unknown.In addition to the above emotional categories,this study includes note-taking behaviors.(3)This study designed an adaptive teaching system framework and implemented an adaptive teaching prototype system based on multi-modal emotion recognition.In view of the flaws that the emotion recognition only stays on the theoretical level at present,this research proposes a common system framework and applies real-time emotion recognition in the real system.And it also designs the personalized aids to highlight the application value of emotion recognition.This paper first introduces the research significance and value of adaptive teaching system based on multi-modal emotion recognition,and then introduces the research status and existing problems of multi-modal emotion recognition,learner behavior analysis and proposes the research route of this paper.Then,this paper describes the behavior modeling of learners and introduces the key technologies such as feature extraction,emotion classification model,data imbalance and adaptive teaching system framework design.After that,the experimental verification of this model and the design and implementation of the prototype system are carried out.Finally,conclusion and future work is given.