The Research And Application Of Pen-holding Gesture Recognition Based On Residual Convolutional Neural Network

The problem of poor pen grip gestures is common among primary and secondary school students,leading to poor writing quality and lack of aesthetics,and incorrect pen grip gestures lead to irregular writing posture,which causes the proportion of myopia among students to rise year after year and hinders calligraphy education and students’ physical and mental health.Based on the video action recognition and intelligent analysis project of our group,this paper firstly designs an asymmetric parallel grip gesture recognition network based on convolutional neural network,deep residual structure and attention mechanism,and then designs and implements a writing online teaching platform based on this network to assist teachers in guiding students to master the correct grip gesture and improve their writing ability.Through the study of domestic and international literature and related technologies on gesture recognition,firstly,a parallel convolutional neural network based on a deep residual structure is proposed for the problem of inadequate feature extraction due to partial occlusion of the gesture in pen-holding gestures,which performs feature extraction on the original image and the gesture segmentation image separately,and then uses the fused features for gesture classification.Secondly,the network is optimized using an improved spatial pyramid pooling block and a hybrid coordinated attention module for the problem of small number of samples in the dataset and varying image sizes affecting feature extraction.Finally,due to the current lack of suitable public pencil gesture datasets,a base dataset containing 7 types of pen-holding gestures with a total of 923 images is constructed in-house and the training set is quadrupled using data enhancement techniques.The experimental results on this dataset show that the proposed network model achieves 76.22% accuracy for pen-holding gesture recognition and82.16% by data augmentation,which has good results and robustness.In order to help students learn to write while continuously improving their pen-holding gestures,this paper designs and implements a prototype writing online teaching platform system based on a parallel convolutional neural network with a deep residual structure.Through this system,teachers publish teaching contents and instruct teaching in the form of courses,and students can learn online independently after selecting courses,while online pen-holding gesture recognition can be performed through photos,videos or cameras as input to assist students in correcting pen-holding gestures.The parallel convolutional neural network based on deep residual structure proposed in this paper can be used to build a pen-holding gesture recognition system,and is also useful for similar gesture classification tasks without high-quality public datasets.The online gesture recognition method and system architecture demonstrated in this paper can also be useful for building other online gesture recognition and teaching systems.