Deep Learning-Based Group Interaction Prediction Via Multiple Granularity Analysis

Group activity analysis refers to the modeling and understanding of multi-participant activities,including the identification of current activity categories and reasoning about future group behavior.It is widely used in video surveillance,security and other fields.In recent years,with the development of individual action analysis techniques,including detection,tracking and posture estimation,researchers have proposed many algorithms and analysis frameworks related to group activity analysis.Inter-individual interaction modeling is a major challenge in group activity analysis.Current work often uses hierarchical framework or graph model to capture the temporal and spatial characteristics of group activities,and then to recognize them.However,there are only a few single-grained predictions on group prediction tasks,such as group trajectory prediction and individual motion prediction.Trajectory and action information describe current activities from different perspectives.Only using one of these information can not comprehensively represent the activity,especially in details.We believe that considering both granularities,i.e.,global and local information,and their interactions,can definitely help activity analysis.Based on the multi-granularity group interaction prediction problem,we predict the multi-granularity information of group activity,including the global position and local posture of each individual.Taking a short video clip with various number of persons as input,we aim to predict a sequence of future skeletal motion data and trajectory for all individuals.First,we use pose estimation method to extract the skeleton of each individual and split it into a center point and other relative points.Then we propose a multigranularity group interaction prediction architecture,which features a single-granularity prediction LSTM(sLSTM)combined with a novel multi-granularity interaction network.This network contains intra-granularity interaction sub-network and inter-granularity interaction sub-network.i.e.,the proposed network focuses on different granularities including trajectory and detailed action.Each sLSTM contains an encoder to capture spatio-temporal continuity.Then to model the interaction between different individuals within granularity,we propose two intra-granularity interaction subnetworks to model interaction in trajectory and action respectively.Further more,to model the interaction between different granularities,an inter-granularity interaction subnetwork based on bi-directional LSTM is employed for its capability of preserving long memory in two directions.Finally,the trajectory and action of each individual are predicted in decoding stage,and the complete skeleton sequence is output through skeleton reconstruction.The proposed method has been comprehensively evaluated on SBU and Choi's New datasets with three evaluation metrics.Experimental results demonstrate that our method can well address group interaction prediction problem.In addition,in order to show the results of group prediction intuitively,we use conditional generative adversarial network(CGAN)to generate the predicted skeleton sequence into the corresponding video sequence.