A Study Of Student Classroom Action Recognition Based On Deep Learning

Human action recognition is an important branch of computer vision technology and has been widely used in many applications.Students’ classroom action is an important part of classroom teaching feedback.Traditional classroom action analysis relies on expert observation,which is time-consuming and inefficient,and the analysis results are subjective.Combining computer vision and machine learning technologies,we can realize the intelligence and automation of classroom action recognition.Compared with traditional RGB images,human skeleton data is more robust to problems such as illumination,occlusion and interference,and is more suitable as a data carrier for classroom behavior recognition.When the existing behavior recognition methods for skeleton data are directly applied to the classroom environment,there is the problem that different classes of actions with high similarity and similar actions with high diversity cannot be effectively recognized.Therefore,in this paper,we focus on the following three aspects of the action recognition method for skeleton data in the classroom environment.1.In order to solve the problem of recognizing different categories of actions with high similarity in classroom environment,the idea of adaptive data preprocessing and acquiring higher-order spatio-temporal features is used to improve the model’s ability to distinguish different types of actions with similarity,and a skeletal action recognition method with adaptive normalization and covariance fusion is proposed.Firstly,an adaptive normalization method is designed to achieve adaptive preprocessing of skeletal data.Secondly,the covariance matrix among skeletal nodes in three axial directions is constructed and filtered with interference information by fusion as descriptors of spatial features of action sequences.Finally,an LSTM network containing adaptive normalization and fused covariance features is constructed to learn the spatio-temporal characteristics of skeletal data and perform classification.Experiments show that the recognition accuracy of the method in this paper is improved by 1.95%,1.10% and 6.33% on the public test datasets Florence,MSR and real classroom datasets,respectively,compared with the current advanced algorithms.2.In order to solve the problem of recognizing same-category actions with high diversity in the classroom environment,the idea of building a domain-adapted learning model with continuous learning capability to achieve the transferability of the model among multiple same-category actions is adopted,and a continuous-learning domain-adapted skeleton action recognition method is proposed.Firstly,the features that can be transferred are strengthened by introducing domain discriminators and attention mechanisms in the deep learning model.Secondly,to make the model meet the demand of automatic updating in online environment,an experience memory-based continuous learning mechanism is introduced to help the model learn new knowledge by simultaneously screening previous experiences in the experience memory when new data are input.Finally,a migration learning framework containing domain adaptation and continuous learning components is constructed.Experiments show that the method has better migration and continuous learning performance on the public test datasets Florence,Kinetics and real classroom datasets,and the accuracy is improved by 2.3%,1.3% and 7.29% on the three datasets,respectively,compared with the current state-of-the-art algorithms.3.A prototype system of classroom action recognition based on skeleton data is designed and implemented to collect students’ classroom behavior data by Kinect sensors,adopt adaptive normalization and covariance fusion of skeleton action recognition method for offline learning,and adopt continuous learning of domain adaptive skeleton action recognition method for online learning,and finally visualize the action recognition and analysis results to help teachers better understand students’ learning status and adjust their teaching methods and strategies in time.There are 30 figures,8 tables and 140 references in this paper.