Deep Reinforcement Learning Based Autonomous Driving Decision-Making Methods

In the field of autonomous driving,decision-making is active and challenging topic currently.Deep reinforcement learning seeks to solve the problem in an end-to-end manner,but generally requires a large amount of sample data and confronted with high dimensionality of input data and complex models,which lead to slow convergence and can not learn effective strategies quickly.Driving strategies are related to a variety of factors,and most deep reinforcement learning based methods use simple constraint reward functions,can only adapt to simple traffic scenarios.Due to the complexity and variability of traffic scenarios,the existing algorithms are less adaptable.To solve these problems,this paper proposes a framework for a driving decision method based on a multi-input multi-factor constrained reward function.The inputs of the method include camera front view,Li DAR data and bird’s-eye view generated from the perception results.Considering the high dimensionality of the input information,two strategies learning methods are designed,and the performance of the methods is evaluated.In the two types of strategies learning methods,the reward functions are all designed with multi-factor constraints such as lateral error,heading,driving smoothness and speed,which can effectively improve the adaptability of the method to the scene and accelerate the convergence speed of strategies learning.The work of the paper mainly includes the following aspects:(1)A driving strategies learning method with multi-sensing input multi-factor constraint(MSMC)reward is proposed.Through analysis the input information characteristics of decision-making,and using the results of environment perception to generate bird’s-eye view,and three types of data are selected as the input of the strategies learning.A strategies learning framework combining Variational Auto Encoder(VAE)and Soft Actor Critic(SAC)algorithm is designed.In order to efficiently use the observed data from multi-sensor inputs,low-dimensional latent features are extract by VAE encoder network and then input to the algorithms to improve the training process speed.The various influencing factors in the decision-making are analyzed,lateral error and azimuth error are defined based on vector field guidance method,and then a multi-factor constrained reward function is designed,which ensure that the strategies can adapt to multiple scenarios.(2)Based on the multi-sensing input multi-factor constraint reward strategies learning framework,the Stochastic Latent Actor Critic(SLAC)algorithm is used to replace the VAE-based representation learning and SAC-based task learning with independent strategies learning methods,and the multi-sensing input multi-factor constraint reward SLAC(MSMC-SLAC)method is proposed.The efficiency of strategies learning is improved by jointly modeling feature representation learning and task learning,which use partially observable markov decision process(POMDP)with probabilistic graphical model and expand ability to adapt to scenarios.(3)The proposed algorithm is simulated and validated.Different traffic scenarios are simulated using the CARLA simulator and visualized using the Pygame software package.The performance of the algorithms is evaluated under different combo of input data;the performance of the proposed method compares with other deep reinforcement learning algorithms;The importance of the multi-factor constraint reward term is experimentally analyzed;Comparison experiments are conducted on the two methods proposed in this paper.The adaptability of MSMC-SLAC algorithm to a variety of traffic scenarios is verified under different simulation maps.Figure 61,Table 15,and 64 references.