Pedestrian Crossing Prediction By Multi-Modal Driven Synthetic-to-Real Transfer Learning

In the traffic scene,pedestrians are the vulnerable group of traffic participants,and their behavioral intention is particularly important for traffic safety.However,the subjectivity of pedestrians is strong,so their behavior intentions also have strong randomness.Among the many behavioral intentions of pedestrians,the most potentially dangerous one is the pedestrian’s behavioral intention of crossing the street,that is,the intention of crossing.Real-time prediction of pedestrian crossing intention can effectively avoid pedestrian traffic accidents during driving.Therefore,pedestrian crossing prediction has gradually become an important research task in driver assistance system or unmanned driving system.At present,the collection conditions of pedestrian crossing prediction datasets are limited,which cannot contain rich scenes and weather conditions,and the data modes are not rich.Therefore,this thesis studies the application of virtual data,constructs virtual datasets for pedestrian crossing prediction,and proposes a pedestrian crossing prediction model based on virtual to real distillation and a pedestrian crossing prediction model based on gated virtual to real knowledge transfer.(1)Due to the limited collection of real data,this thesis uses CARLA simulator to synthesize virtual video sequence about pedestrian crossing offline,and constructs virtual datasets Virtual-Pedcross-4667 and S2R-PCP-3181,which contain 4,667 and 3,181 virtual video sequences respectively.Compared with Virtual-Pedcross-4667 dataset,S2R-PCP-3181 dataset contains more data modes,which can give full play to the advantages of virtual data.Virtual datasets can be used to help prediction models more fully learn pedestrian crossing rules from real data.(2)Although pedestrian crossing knowledge in virtual data can be used as reference for real data,there are large domain differences between virtual data and real data.Therefore,solving the problem of domain differences is the key to handle the cross-domain data.After generating a lot of virtual data,this thesis designs a pedestrian crossing prediction model based on virtual to real distillation.The special feature of this model is that it uses the idea of knowledge distillation and the teacher model trained on the virtual dataset to guide the learning of the lightweight student model on the real data,so as to improve the predictive performance of the student model and provide the foundation for the deployment and application in the future.(3)Because the transfer method adopted by the aforementioned model is relatively simple,the proper domain transfer method is not selected to process the multi-modal data with different domain differences,and the role of different modal data cannot be given full play.Therefore,this thesis constructs a pedestrian crossing prediction model based on gated virtual to real knowledge transfer.In this model,different knowledge transfer methods are designed for different modal data,and the weight of each method is learned adaptively by a learnable gating unit,so as to achieve the optimal performance.Experiments were carried out on JAAD and PIE real data sets.The pedestrian crossing prediction model based on virtual to real distillation achieved 86% and 89% accuracy,respectively,while the model based on gated virtual to real knowledge transfer achieved 88% and 92%accuracy,respectively,which improved by 2% and 3% compared with the optimal benchmark model.This shows that the model proposed in this thesis can correctly transfer the virtual and real information,so as to correctly predict the pedestrian crossing intention.The pedestrian crossing prediction model proposed in this thesis can provide algorithm ideas for pedestrian perception and collision avoidance modules in driver assistance system and unmanned driving system.If intelligent vehicles can predict pedestrians’ crossing intentions in real time and make correct decisions and controls in time,pedestrian traffic accidents can be effectively avoided.