| Action recognition investigates human behavior based on spatial appearance and temporal dynamics information in video sequences.As an active research in computer vision field,it has been widely applied in intelligent monitoring,virtual reality,human-computer interaction,etc.Two-stream network based approaches are popular for action recognition,consisting of appearance branch and motion branch.While the former branch takes RGB images as input,the input modality of the latter is optical flow.Each branch firstly performs action recognition independently and the outputs of both branches are fused subsequently as the final prediction.However,a major problem is the heavy computational cost involved in optical flow computing,which has become the bottleneck of two-stream methods for practical application.Aiming at this issue,this paper makes exploration from two aspects: Cross-Modality Distillation and Motion Mimic.The main research contents are as follows:(1)Cross-Modality Distillation for Action RecognitionThe optical flow branch in two-stream network has better motion representation.According to transfer learning strategy in knowledge distillation,the optical flow branch is chosen as the teacher model to supervise the training of RGB branch(i.e.,student network).The trained student model combines both appearance and motion information,named as Appearance and Motion RGB Stream(AMRS).Moreover,since the original RGB branch can provide complementary information,a new network is built by combining the original RGB branch and AMRS to further improve the recognition performance.This new network is named appearance and motion fusion network(AMFNet).We design three distillation schemes as alternatives,and experimentally choose feature-based knowledge distillation for action recognition.In this scheme,by minimizing mean square error(MSE)loss between the high-level features of both the teacher network and the student network,motion information in the teacher network(the optical flow branch)is transferred into the student network(AMRS).In the proposed AMRS and AMFNet,optical flow sequences are utilized only in the training phase,not involved in the test stage.As only RGB images are taken as inputs in testing,AMRS not only reduces the computational cost and time delay significantly,but also facilitates the acquisition and processing of input modality in practical applications.Experiments are conducted on two commonly used datasets,UCF101 and HMDB51,verifying the effectiveness of the proposed algorithm.(2)Action Recognition Based on Motion SimulationIn this chapter,we further explore how to obtain motion representation without optical flow.It is observed that optical flow pays more attention to the contours of the target and edges of the scene.Taking computation of optical flow as reference,we design a motion simulation module(MSM)for motion information extraction,provided only RGB sequences are available as input.Since the edge and contour information is mainly present in low-level feature maps,the difference of the target entity between adjacent frames in these feature maps is utilized to extract motion features.Here we design three schemes for motion information extraction,including feature difference,position difference and joint feature difference,respectively.The motion features extracted by MSM are input into the backbone network for training.The trained motion simulation module(MSM)is combined with the original appearance RGB branch,constituting the motion simulation fusion network(MSFNet)for final action recognition.In this paper,we employ Res Net50 as the backbone network.On two popular datasets,UCF101 and HMDB51,experimental results have demonstrated the effectiveness of the proposed algorithm. |
