A Study Of Student Classroom Behavior Recognition Based On Domain Adaptation And Continual Learning

Students’ classroom behavior is an important part of classroom teaching feedback.Traditional classroom behavior analysis relies on expert observation,which is timeconsuming,inefficient and subjective.Intelligence and automation of classroom behavior recognition can be achieved by combining computer vision and machine learning technology.Compared with traditional RGB images,human skeleton data is more robust to light,occlusion,interference and other issues,and is more suitable for classroom behavior recognition.However,when the existing behavior recognition methods for skeleton data are applied directly to the classroom environment,there are problems such as the lack of data on skeleton behavior in the classroom environment,the high labeling cost and the difficulty of fine-grained behavior recognition.Therefore,in view of the behavior recognition methods of skeleton data in the classroom environment,this thesis conducts research work from the following three aspects:(1)To solve the problem of scarce data and high labeling cost of classroom skeleton behavior,which is difficult to meet the needs of deep learning,a skeleton domain adaptation behavior recognition method STT-DA(Spatial-Temporal Transformer based Domain Adaptation)based on spatial-temporal Transformer is proposed using the domain adaptation learning framework.Firstly,normalize and align the source and target domain data,then divide the skeleton sequence data into subsequences to reduce model complexity and pay attention to sub-actions;Secondly,based on Transformer’s self-attention and cross-attention mechanisms,a domain adaptation learning method containing three stream data was constructed.This method considers the characteristics of skeleton data and simultaneously aligns features in temporal and spatial dimensions;Finally,in order to verify the effectiveness of the domain adaptation method proposed in this chapter and its application effect in classroom environments,experimental validation was conducted on public datasets and self-collected Classroom dataset.The results showed that the recognition accuracy reached 82.5% on NTU → UCLA,and 36.6% on NTU →Classroom.Compared with other domain adaptation methods,the accuracy has been improved to varying degrees.(2)To solve the problem of fine-grained action recognition in continuously changing classroom environments,a continual learning method(DGS-FA)based on the experience sample selection strategy DGS(Double Gradient sample Selection)and the skeleton data multi view enhanced FA(Feature Augment)strategy is proposed on the basis of domain adaptation methods.A small-scale labeled classroom dataset is applied to continuously and dynamically adjust the parameters of the domain adapted behavior recognition model to alleviate the catastrophic forgetting problem during the continual learning parameter update process.It improves the classification and recognition accuracy of classroom behaviors among students with intra class diversity and inter class similarity.The experiment shows that the application of continual learning greatly improves the accuracy of classroom behavior recognition,reaching90.3%.The DGS and FA strategies also reduce the average forgetting rate.Combining the architecture of domain adaptation and continual learning methods,on the one hand,it alleviates the requirements of deep learning on data volume,on the other hand,it enhances the generalization ability of the model.Compared with the traditional supervised learning methods,the architecture proposed in this thesis improves the accuracy of classroom behavior recognition on small batch datasets and has widely applications.(3)Based on the research on behavior recognition methods for student skeleton data proposed in this thesis,a classroom behavior recognition prototype system was designed and implemented.The system utilizes Kinect sensor devices to collect students’ behavior information during classes,and combines continual learning with domain adaptation to achieve continuous learning of behavior characteristics during class and dynamically adjust the classification model.Finally,the behavior recognition results are visualized and displayed.