Research On Deep Reinforcement Learning For Urban Adaptive Traffic Signal Control

With the growth of population and the acceleration of urbanization,the demand for urban travel has increased greatly,and the existing transportation infrastructure is unable to meet the increasing demand for urban travel,resulting in the cyclical and non-cyclical congestion in the city.Traffic signal control is the core of urban traffic management and control.A scientific and reasonable signal control scheme can maximize the throughput of intersections,improve the operation efficiency of urban road networks and the capacity of intersections,and reduce the frequency and intensity of certain types of traffic conflicts.Thereby alleviating the problem of urban traffic congestion.The traditional traffic signal control method has limited adaptability to the dynamic traffic environment.Although the adaptive traffic signal control system can alleviate the traffic congestion problem in the city to a certain extent,it is mainly based on the prediction of the fixed traffic model and the preset signal control scheme,or based on real-time traffic simulation model to predict the traffic demand.Its essence is based on the traffic model driven,and it is necessary to calibrate the parameters of the traffic simulation model and design the pre-defined signal control scheme according to the actual traffic scene in advance.However,due to the actual traffic environment has the characteristics of dynamics,randomness and uncertainty,and the applicability of simulation model parameters and preset signal control schemes has the poor applicability.Traffic signal control based on deep reinforcement learning has the characteristics of model-free,self-learning,data-driven,online closed-loop control,etc.,which can effectively overcome the disadvantages of the adaptive traffic signal control system.Therefore,this paper aims to establish a single intersection signal control model and the arterial intersection signal coordination control model based on deep reinforcement learning algorithms,and realize the coordination of the arterial intersection traffic signal control through the perception state interaction between agents and the reward information sharing mechanism.The main content of the paper includes the following aspects:First,systematically discuss urban traffic signal control strategies and emerging deep reinforcement learning algorithms,analyze the advantages and disadvantages and applicability of different signal control strategies and deep reinforcement learning algorithms,and elaborate on the control mechanism of each deep reinforcement learning algorithm.Lay a theoretical foundation for the subsequent chapters based on the deep reinforcement learning algorithm for single intersection and arterial intersection signal control models.Secondly,establish a single intersection signal control model based on deep reinforcement learning,and design the traffic state,action control parameters and reward function of the agent in the single intersection traffic signal control model for different deep reinforcement learning algorithms.The single intersection training scene is established through SUMO microscopic simulation software,and the influence of the design parameters of the agent,the training parameters of the reinforcement learning,and the parameters of the traffic simulation model on the performance of the algorithm is analyzed.In addition,the average vehicle delay,the maximum queue length and the number of vehicles passing through the intersection are selected as performance evaluation indicators,and the proposed single intersection signal control model and timed signal control are compared.Thirdly,the arterial intersection signal control model based on deep reinforcement learning is established,and the joint traffic state,action control parameters and joint reward function of the agent in the arterial traffic signal coordination control model are designed.The SUMO micro-simulation software is used to establish the arterial intersection training scene,and analyze the influence of the design parameters of the agent,the training parameters of the reinforcement learning,and the parameters of the traffic simulation model on the performance of the algorithm.In addition,the average vehicle delays,the number of stops,and the driving speed are selected as performance evaluation indicators,and the proposed arterial intersection signal control model and the arterial green wave coordinated control are compared.Finally,the actual traffic demand data of Hangzhou is used to test the robustness of the single intersection signal control model and the arterial intersection signal coordinated control model under demand disturbance scenarios.The average vehicle delay is used to describe the signal control efficiency and the standard deviation of vehicle delay is used to describe the stability of signal control under demand disturbance conditions to construct a robustness index of signal control.In addition,the arrival flow rate and average headway of each lane per unit time are selected as the calibration parameters of the simulation model,and the validity of the calibration results is tested.Finally,under actual demand disturbance conditions,the proposed signal control model is compared with the benchmark signal control scheme to verify the robustness and effectiveness of the proposed model.