Research On Multi-Modal Data Fusion And Multi-Task Prediction Methods For Environment Perception Of Autonomous Driving

Recent advancements in computer vision have revolutionized the automotive industry,enabling the emergence of autonomous vehicles that can efficiently address traffic-related issues such as traffic congestion,road accidents,and environmental pollution.Perception plays a critical role in enabling autonomous vehicles to sense their environment and predict the state of moving objects for proactive collision-free trajectory planning.This thesis investigates three key visual perception problems for autonomous driving,including traffic object detection,lane-line recognition,and drivable/nondrivable area identification.The main objective of the thesis is to develop a comprehensive visual perception system which is capable of understanding complex environments and dynamic scenes for safe and efficient autonomous driving.The contributions of this thesis are summarized as follows.It proposes a LiDAR-based framework for moving object detection and speed estimation,which uses Model Predictive Control(MPC)to predict a controlled speed and a collision-free trajectory.The object detector detects moving objects whose centroid attributes are tracked using Interactive Multiple Model-Unscented Kalman Filter-Joint Probabilistic Data Association Filter(IMM-UKF-JPDAF).Then tracking data are used to estimate the speed,distance,and direction of the moving objects.It also introduces a Behavior Prediction Algorithm(BPA)that observes the motion states of the moving objects using tracking data and determines an optimal and collision-free trajectory.Simulation results demonstrate that the proposed framework is a feasible solution to observe the traffic conditions and find a collision-free trajectory in a dynamic environment.It presents a point-pixel early-fusion algorithm for efficient object detection and classification with depth information in autonomous vehicle systems.The point-pixel fusion transforms 3D LiDAR points into 2D-pixel frames,and similarly,the camera image is also transformed into 2D-pixel frames in order to create a 2D representation of the 3D environment.Subsequently,YOLOv4 is used to detect traffic objects as the region of interest(ROI),and then the LiDAR points are fused into the ROI to determine the depth of the object detected by YOLOv4.Only the LiDAR points in the ROI are retained for depth estimation,and the rest of the LiDAR points are repudiated.The processing of LiDAR data remains challenging due to the uneven distribution and sparsity of LiDAR point clouds.The proposed ROI-based early fusion approach processes only the LiDAR points that are in the ROI,and can reduce the computation cost of the fusion process.The experimental results show that the proposed ROI-based early sensor fusion method is an efficient and reliable solution for object detection,classification,and depth estimation for an autonomous driving system.It designs an enhanced encoder-decoder multi-task framework,which can jointly perform three essential visual perception tasks,including object detection,drivable area segmentation,and lane line identification.The proposed architecture consists of a shared encoder and three independent decoders,each of which is responsible for a specific task.The proposed architecture is compact and adopts an end-to-end training paradigm,which yields higher accuracy and faster inference speed.Experimental results demonstrate the excellent performance of our proposed architecture on the challenging BDD100K dataset.Our proposed architecture is also computationally efficient,achieving a real-time inference speed of 43 FPS on an Nvidia GTX TITAN X-12GB GPU.