Research On Multi-modal Multi-task End-to-end Autonomous Driving Method Based On Visual Perception

Since deep learning,especially convolutional neural networks,has achieved remarkable success in the field of computer vision,autonomous driving technology based on computer vision and deep learning has developed rapidly.The current research methods of the end-to-end autonomous driving system mainly take an image or an image sequence as input,and directly predict the steering wheel angle with convolutional neural network,which has achieved good results.But it is not sufficient for self-driving vehicle control by taking the advantage of the steering wheel alone.Therefore,we propose a multi-modal multi-task neural network model based on end-to-end learning,and predicts the steering angle and speed of the vehicle to realize the horizontal and vertical control of the autonomous vehicle.The main research contents are as follows:Based on the GTAV virtual game,a simulation platform suitable for autonomous driving data collection and testing is established.The simulation platform can effectively shorten the algorithm development time and reduce the development cost,and can intuitively test the effect of the algorithm model.In order to solve the horizontal and vertical control problem of autonomous driving,we take the previous steering wheel angle sequence and speed sequence as extra modality,and propose CNN-LSTM multi-modal multi-task neural network model to predict the steering wheel angle and speed simultaneously in an end-to-end manner.The neural network model is trained and tested on the GTAV dataset.The experimental results verify the feasibility and effectiveness of the CNN-LSTM multi-modal multi-task neural network model for the horizontal and vertical control of the autonomous vehicle.It shows that the model provides an accurate speed prediction while further improves the state-of-the-art steering angle prediction.The interpretability of the neural network model is poor.In order to explain to some extent how the CNN-LSTM neural network model understands the input image and find the direction of the model improvement,visualizing the training of the network with the VisualBackProp method.The result shows that the CNN-LSTM network can extract useful features for autonomous driving by itself.Real-world scene data collection and road testing were performed through a ROS-based autonomous driving platform.Using transfer learning to solve the difference between the GTAV dataset and the real-world scene dataset,the retraining and testing of the neural network model is performed on the real scene dataset using finetune.The test results show that the model can accurately predict the steering wheel angle and speed value.The road test results show that the neural network model can better complete the task of keeping lane and avoiding obstacle.