Research On Deep Reinforcement Learning Driving Decision-making In Roundabout Environment

With the rapid development of computer technology,automatic driving has become the mainstream direction of the development of the automotive field,and decision planning is one of the key technologies.How to ensure the safety of vehicles in highly interactive urban scenes(such as roundabouts,intersections,etc.)has become an important issue in the development of autonomous driving technology.Compared with rule-based methods that require a specific scenario-specific design,Deep Reinforcement Learning(DRL)algorithms have greater advantages.DRL avoids the heavy algorithm design process,and the performance can be improved through continuous training.This paper aims to apply deep reinforcement learning to the decision-making scene of autonomous driving around the island to study the decision-making problem of intelligent vehicles in complex scenes.This article first introduces the relevant knowledge of reinforcement learning,including basic concepts,deep reinforcement learning algorithms,and offline reinforcement learning algorithms.Then the particularity of the roundabout decision is analyzed,including the stages of entering the roundabout,driving inside the roundabout,and driving out of the roundabout.At the same time,the Carla simulator is introduced,and a reinforcement learning training environment is built in combination with the reinforcement learning gym framework.Afterward,the global path is planned using the~*A algorithm,and a discrete point smoothing algorithm based on the principle of quadratic optimization is designed,and the smoothed path is used as the navigation path for subsequent simulations.Then,according to the characteristics of driving around the island,a decision-making algorithm based on deep reinforcement learning,rule-based and fusion of the two is designed.The decision-making algorithm based on deep reinforcement learning adopts the Soft Actor-Critic(SAC)algorithm as the basic algorithm and combines the Two Time Scale Update Rule(TTUR)and the Prioritized Experience Replay(PER)algorithm integrated into the SAC algorithm,the TUPE-SAC algorithm is proposed,and the algorithm update process and network architecture are designed.At the same time,the state space is designed for different stages of round-the-island driving;in the action design,the brake and accelerator are combined into acceleration control,and frame skipping is used to simplify the decision-making difficulty;the reward function is designed according to the expected vehicle driving behavior and round-the-island characteristics.In the last chapter,the simulation training and verification of the above algorithm are carried out.The results show that the training speed of the TUPE-SAC algorithm has a good improvement compared with other algorithms such as SAC,and optimal driving safety,better driving comfort and driving efficiency have been obtained in the verification stage.In this paper,rule decision-making is divided into vertical decision-making and horizontal decision-making.Longitudinal Decisions Control both speed and steering within a roundabout.Among them,the speed is controlled by the Intelligent Driver Model(IDM)which is improved according to the characteristics of driving around the roundabout,the steering is controlled by the pure pursuit model,and the last two are output through PID feedback.The lateral decision is the lane change decision.The Minimizing Overall Breaking Induced by Lane Changes(MOBIL)is used to identify the lane change intention,and the quintic polynomial curve is used to generate the lane change trajectory.To further improve the safety of decision-making,a fusion decision-making framework is proposed,and Implicit Q-Learning(IQL)is used to evaluate different policies.In order to improve the accuracy of the evaluation,different trajectory sets are used to train and test the IQL algorithm.The test results show that the IQL algorithm based on the mixed trajectory set has the best performance among all IQL algorithms,and it is used in subsequent fusion decisions.Finally,the simulation experiment of fusion decision-making is carried out.Based on the well-trained TUPE-SAC algorithm,rule decision-making and IQL algorithm,two Light Fusion(LF)decision-making algorithms,LF-SAC-Rule and LF-SAC-IQL,are proposed and verified by simulation.The results show that LF-SAC-IQL obtains slightly better driving safety.At the same time,in order to improve the safety of the vehicle training process,the RF-SAC-IQL algorithm based on the Re-Fusion(RF)framework is proposed,and it is trained and verified in the simulation environment.The results show that the refusion framework can greatly reduce the output of dangerous actions in the action exploration stage of DRL,ensuring the safety of the vehicle training process and reducing the training cost.In the final test,it also obtained the better safety,driving efficiency and driving comfort.