Rule-Constrained Reinforcement Learning For Intelligent Vehicle Motion Control At Unsignalized Intersections

Autonomous driving technology is one of the areas that have received much attention in recent years,and many companies and research institutes at home and abroad have invested a lot of resources in related research.As an important traffic scenario,intersections play a crucial role in the efficiency and safety of the whole traffic.Controlling autonomous vehicles to complete unprotected left turns at intersections is a challenging task.Due to the considerable uncertainty and unpredictability of the traditional rule-based autonomous driving planning control algorithm,it is difficult to construct an accurate and reliable mathematical model for such complex scenarios.The unprotected left turn problem is usually characterized by high latitude and nonlinearity,and deep reinforcement learning combines the advantages of the powerful function fitting ability of deep learning and the decision-making ability of reinforcement learning,revealing better experimental results than traditional planning and control methods.Therefore,this paper proposes a rule-constrained reinforcement learning planning and control method for autonomous driving under complex urban road conditions,focusing on left-turn conditions at unsignalized intersections,based on the "NSF Joint Foundation Key Project".The innovative work of the paper includes the following aspects:(1)An autonomous driving planning algorithm combining A* algorithm and DWA algorithm is designed.In order to avoid obstacles effectively,the improved DWA algorithm is chosen as the local path planning algorithm in low-speed state in this paper.A TTC-based dynamic obstacle avoidance mechanism is designed,which can optimize the collision detection range and response time to further improve the safety of the trajectory.Meanwhile,in order to avoid the generated trajectory from falling into local optimum,the result of global planning algorithm A* is designed as a trajectory evaluation subfunction to ensure that the autonomous vehicle can travel along the overall path and yet adapt to the random situation.(2)A rule-constrained reinforcement learning intelligent vehicle control method is designed.The results of the trajectory planning module are introduced into the framework of reinforcement learning as a target condition to train a reinforcement learning controller with rule constraints.The control policy is learned by interacting with the environment,which does not rely too much on the model accuracy of the controlled object,while the state input takes into account the vehicle kinematic characteristics,which is more secure and reliable than the end-to-end learning approach.A target-guided reward function that takes into account the heading error and other factors is designed to accelerate the learning of the control strategy.The average reward of episodes after network convergence is also higher than the training result of DDPG algorithm by more than 30%.(3)Simulation scenarios covering a variety of typical left-turn conditions are designed to test the effectiveness of the proposed method.A two-way four-lane intersection based on the CARLA simulator was built to randomly generate social vehicle traffic flow to ensure the robustness of the rule-constrained reinforcement learning planning control algorithm.Experimental results show that the proposed method is able to control the autonomous vehicle to reliably perform the unprotected left-turn task with about 23% improvement in the passing success rate while ensuring similar driving efficiency compared to the rule-based and end-to-end control methods.Finally,the co-simulation of CARLA and ROS using Carla-Ros-Bridge effectively reduces the gap between simulation and reality,and provides theoretical and technical support for subsequent real-world deployment.