Research On Ramp Merging Decision-Making For Autonomous Driving Based On Deep Reinforcement Learning

With the continuous maturation of autonomous driving technology,ensuring the safe and stable operation of autonomous vehicles in complex scenarios is a critical issue that needs to be addressed.As a common and important section of the road,the entrance ramp plays an important role in improving the overall safety of the road by ensuring the reasonable operation of autonomous vehicles in this section.In the process of autonomous driving ramp merging,rule-based methods are often limited to some specific scenarios,while deep reinforcement learning methods can achieve better performance in complex dynamic scenarios through continuous learning.Therefore,this study researches on the decision-making behavior of autonomous vehicles in the ramp merging process based on deep reinforcement learning methods.The specific research contents are as follows:(1)Establish autonomous driving ramp merging decision-making models based on two reinforcement learning algorithms,namely,deep Q-network algorithm and deep deterministic policy gradient algorithm.The state space includes observed values of the motion states of the ego vehicle and surrounding vehicles.Considering safety,efficiency,and comfort as the multiple optimization objectives that need to be satisfied during autonomous driving,the reward functions in the two decision-making models are designed based on safety,efficiency,and comfort.Subsequently,the SUMO simulation software is used to construct the autonomous driving ramp merging scene,and the two ramp merging decision-making models are trained and verified through simulation.Finally,a comprehensive comparison and analysis of the two models are conducted based on various aspects such as the total average reward obtained by the ego vehicle during the ramp merging process,stability,efficiency,comfort,and practical significance.The results indicate that the ramp merging decision-making model based on the deep deterministic policy gradient algorithm has better comprehensive performance and practical significance.(2)By establishing an intelligent driver model and adjusting the parameters of the model,the aggressive and conservative driving styles of the surrounding vehicles on the main road are characterized.Considering the different pursuit of efficiency,comfort,and safety distance during the driving process of vehicles with different driving styles,two autonomous driving ramp merging decision-making models are respectively established for aggressive and conservative driving styles,by adjusting parameters such as the comfort weight,efficiency weight,and expected safety distance in the reward function.Both models are trained and verified under the scenario of different driving styles of surrounding vehicles on the main road,and the verification results are compared and analyzed.The results show that when the surrounding vehicles on the main road have an aggressive driving style,both ramp merging decision-making models of different driving styles perform more frequent acceleration changes and higher speeds in the merging area to find the appropriate merging opportunity.Meanwhile,the ramp merging decision-making model of the conservative driving style shows a lower average absolute jerk value,indicating that the vehicle with a conservative driving style performs smaller changes in acceleration and achieves a higher level of comfort.Overall,based on all the results,the two different ramp merging decision-making models established for different driving styles can achieve the ego vehicle’s different pursuit of driving goals.(3)Different traffic flow densities on the main road may have an impact on the performance of autonomous vehicles in completing the ramp merging task.Therefore,an autonomous driving ramp merging decision-making model that considers traffic flow density is established,and the model is trained accordingly.The performance of the ego vehicle in ramp merging is then validated and compared under different traffic flow densities.The results show that within a certain range,as the traffic flow density decreases,the average speed of the vehicle throughout the driving process and merging area gradually increases,and the average merging time gradually decreases,indicating an improvement in driving efficiency.