From Virtuality To Reality:Research On Deep Reinforcement Learning Based Autonomous Vehicle Control

With the increasing pressure of environmental pollution,road accident and traffic jam,Intelligent Connected Vehicle(ICV)has become a hotspot in the modern automobile industry as well as a main direction and key competition area for vehicle development in the future.New demands are needed to be satisfied for autonomous vehicle control under this trend.The autonomous vehicles traveling in different scenarios need to efficiently and steadily perceive and understand the current environment and make rational decisions.Currently,intelligent vehicles can do planning,decision making and control well in simple driving scenarios and urban road structure;however,human intervention is still needed in complex driving scenes.For example,it is reported that Waymo – the most advanced intelligent vehicle now in the market,can hardly cut into the normal traffic flow during left turn at a T-junction in Phoenix City;sometimes even the simple right turning can be a trouble problem when there are other traffic participants around the corner.One possible reason for this phenomenon is that the test and validation for autonomous driving under complex scenes is always hard to perform in the real world because of the safety concern;for another,the autonomous vehicle control methods are mostly rule-based(i.e.,the vehicle behavior under every possible case is hand-coded),that means the rule number in the complex driving scene will increase exponentially and different rules may be conflict.Optimal control method with constrains can also be a solution for the common driving case,but when it comes to the complex driving scenarios,things could be urgly.There may be only local optimum or even no slolution for the optimal problem,so the vehicle has to stop there waiting for all safety concerns disappear.To address these problem,intelligent control should be applied so that plenty amount of data can be directly used or obtained from the interaction between agent and environment.By learning from plenty amount of data,the agent can be self-learning and self-adaptive to fit into the real-world applications.Reinforcement learning(RL)is a typical experience-driven and self-learning method which has achieved great progress in robot,drones and vehicle areas.However,due to the inherent storage,computation and sampling complexity,it’s greatly limited to deal with only low-dimentional features.Recently,deep neural network with nonlinear fitting and feature representation capalibities provide a new idea for solving this problem.The emergence of Deep learning(DL)has had a profound impact on various areas,like object detection,scene sementic segmentation,speech recognition and language translation.The most important feature of DL is its ability to automatically find the low-dimentional representations of high-dimentional data.DL also promoted the devepolment of RL,by combining these two methods together,deep reinforcement learning(DRL)was developed and used to deal with high-dimentional state space and discrete/continuous action space in decision-making domain.The deep part is like our human eyes which is responsible for perception and feature extraction;the reinforcement part is like our brain and responsible for reasoning,judgement and decision-making.By interacting with the driving environment,DRL can learn how to obtain the features and how to behave in complex driving scene,making it a viable solution for fully autonomous vehicle in various scenarios.Currently,most of the DRL-based autonomous driving control methods take only images as input,this always result in a insufficient state representation;meanwhile,the DRL agent is usually trained in virtual simulation platform,the huge gap between virtual and real scenarios inhibit the trained agent from generalizing to the reality.With the fund of the National Key Research and Development Program of China – “Theory of Human Machine Interaction in Intelligent Electric Vehicle”(No.2016YFB0100904),the National Natural Science Foundation of China – “Research on Integrated Modelling and Control of Intelligent Electric Vehicles”(No.U1564211)and National Key Research and Development Program of China –“Research on simulation environment modelling and Hardware-In-the-Loop Testing technology for autonomous vehicle”(No.2018YFB0105103),in this paper,we propose a DRL control method which can be transferred from virtual training environment to real application.In this method,we aim at developing a DRL network that can effectively extract features from the heterogeneous data and is robust to various driving conditions.To effectively get the representation of the state space from visual input,we establish a comprehensive visual scene understanding framework which includes lane detection,vehicle 3D pose estimation and sementic segmentation of driving scenes.By taking the result of visual scene understanding framework as input,we design the DRL control system and make it generalize from virtual training scene to real driving application.When given a destination,the intelligent driving agent can extract useful information from visual input and control the car autonomously under the global navigation guidance,different tasks like turning,overtaking,lane change,and obstacle avoidance need to be finished.Based on the theoretical research and virtual training,DRL control scheme could be validated on the experimental vehicle platform in real world.The main research content is as following:(1)Establishing a comprehensive visual scene understanding framework.The visual scene understanding is based mainly on semantic segmentation and takes lane detection and 3D vehicle pose estimation result as supplement.The lane detection algorithm combines traditional hand-coded feature-based method and DL-based method together to obtain real-time and accurate lane line detection result which in turn providing accurate relative vehicle positioning.The 3D vehicle pose estimation leverage the DL capability and geometric constrains to estimate vehicle pose from monocular image,which provides a real-time position and attitude information of surrounding vehicles.As to the semantic image,it is the source of implicit information input of our introduced DRL network.(2)Research on novel dilated-residual deep neural network(DNN)for semantic segmentation.Semantic segmentaion can remove the detail information of the image yet still preserve the semantic relationship between objects,which makes it primary part of the scene understanding and the medium for the DRL to generalize from virtual training scene to real world application.To overcome the problem of gradient vanishing or gradient exploding,we introduce the identity mapping concept into encoder-decoder of the semantic network.To solve another problem that the continuously downsampling in DNN harms the small object,dilated convolution is deployed in the designed network.In order to avoid the impact of the gridding phenomenon caused by dilated convolution,we propose to use hybrid dilated structure to fully extract the information.Meanwhile,residual operation is eliminated at the end of the network so that the gridding data is blocked thoroughly.(3)Research on visual scene understanding based DRL control scheme.Firstly,we design the deep deterministic policy gradient(DDPG)network by considering the heterogeneous data input,different feature extraction units(FEUs)are proposed to extract features from different dimensional data,dropout is used between FEUs to make the data missing so as to improve the robustness of the trained driving agent.To enable the agent to complete all the tasks and improve the convergence rate,different influential factors,like vehicle speed,steering angle,lane departure factor,collision,are taken into consideration when designing the reward function.Terminal conditions are aslo set to guarantee the convergence speed and avoid collecting much invalid data sample.Finally,the mainstream virtual simulation platforms are compared,and CARLA is chosen for training the agent,tasks like going straight,turning left and right,cruising in the traffic flow are set to train and test the DRL agent.(4)DRL control method validation in vehicle platform and real driving conditions.For validation our method in reality,firstly,we choose a vehicle platform and refit it.Three cameras are mounted and calibrate,the electronic and electrical system are designed,control and data acquisition systems are redesigned.For obtaining a camera view matching with the virtual scene,online registration is carried out.For safety concern,we take semi-closed campus scenes as test scenes,three testing conditions are designed,i.e.,going straight,turning right and obstacle avoidance.To avoid any possible crash,we validate the algorithm under the scenario without too many traffic participants.Some challenging tasks are also tested and domenstrated in demo video.After the test,the behavioral rationality of three tasks are analyzed.To sum up,the major innovated idea proposed in this paper are as follows:(1)A comprehensive visual scene understanding framework is proposed,providing a good feature representation basis for DRL control.Considering that the simple semantic segmentation cannot accurately include the precise vehicle positioning and vehicle pose information,real-time lane detection and vehicle pose estimation algorithms are proposed to supplement the semantic information.The lane detection algorithm combines traditional hand-coded feature-based method and DL-based method together to obtain real-time and accurate lane line detection result which in turn providing accurate relative vehicle positioning.The 3D vehicle pose estimation leverage the DL capability and geometric constrains to estimate vehicle pose from monocular image,which provides a real-time position and attitude information of surrounding vehicles.This combined visual understanding information can effectively increase the DRL model convergence speed.(2)A novel dilated-residual deep neural network(DNN)for semantic segmentation is proposed.In semantic segmentation task,dowmsampling is always operated to increase the receptive field at next layer,but the small object may miss accordingly.In this paper,we innovatively introduce the dilated convolution and identity mapping concept in residual network into our designed network to address this problem.To make up the gridding phenomenon caused by dilated and residual network,hybrid dilated convolution structure is applied in the asymmetric residual module of the network and the last two layers of the encoder are designed as un-residual to block and eliminate gridding features.Therefore,the DNN representation capalibity will not decrease as the depth of the net increase and the gridding is avoided.The new structure effectively reduces the network size,improves the computational efficiency and guarantees the segmentation accuracy.(3)A DRL based control method that can be trained in virtual scene and generalized into reality without any prior knowledge is proposed.For obtaining a good feature representation from the visual scene understanding result,various feature extraction units(FEUs)are proposed to extract features from different dimensional data,dropout is used between FEUs to make the data missing so as to improve the robustness of the trained driving agent.To enable the agent to complete all the tasks and improve the convergence rate,different influential factors,like vehicle speed,steering angle,lane departure factor,collision,are taken into consideration when designing the reward function.To validate the trained agent,a vehicle plarform is established and several experiments in the real world is carried out.Because of the good performance of visual scene understanding part and the training design,the vehicle can well complete going straight,turning and obstacle avoidance tasks in the real world.