Research On Student Behavior Description Methods In Mainstream Learning Scenarios

As an interdisciplinary research topic of Computer Vision,Multimedia,Artificial Intelligence and Natural Language Processing,image captioning is an effective method to describe image content automatically with one or more sentences,which makes it possible to intelligently analyze student behaviors in learning scenario in modern education.In this dissertation,the behavior analysis and description of college students is taken as the research object,and two relevant datasets suitable for describing student behaviors in learning scenarios are constructed with reference to some existing image description datasets.On this basis,we suggest several new methods and techniques for describing student behaviors in different scenarios.Our main innovative work of this dissertation includes:(1)According to the behavior characteristics of students in standardized examination room,"ExamineeActivityCaptioning Dataset",the dataset of student behavior description is constructed.On this basis,a single sentence description method on student behavior based on CNN+LSTM network in the examination scenario is proposed.This method firstly employs CNN to automatically obtain the behavior characteristics of a student,then uses LSTM network to transform the behavior characteristics into a sentence to describe the behaviors of a student.Our experimental results show that the suggested method can correctly describe writing,turning over the test paper,looking around and other 5 kinds of student behaviors in the examination scenario.(2)According to the characteristics of student behavior in traditional classroo:s and laboratories,a dataset of student behavior dense descriptions in learning scenarios,"LearnerActivityDenseCaptioning Dataset",is constructed.On this basis,a corresponding dense description method based on CNN+LL+LSTM network is proposed.In this method,CNN is firstly used to automatically acquire the student behavior characteristics.Then,a LL network is employed to detect the possible behavior regions.Thirdly,a LSTM network is used to transform every regional features into a single sentence for describing a behavior of certain a student,which results in a sentence sequence.Namely,dense descriptions on student behaviors are generated.Our experimental results show that the method can accurately describe 14 kinds of student behaviors including listening,raising hands,and sleeping in the common learning environment in classrooms or laboratories.(3)To shorten the description time for an image,YOLO v3 model is discussed,and a fast and effective dense description method on student behavior based on YOLO+LSTM network is further proposed.Our experimental results show that compared with the dense description method based on CNN+LL+LSTM network,the proposed method not only can effectively densely describe 14 kinds of behavior regions such as listening,raising hands and sleeping,but also can reduce the testing time of single image from 240ms to 150ms.