Research On Freeway Ramp Metering Method Based On Deep Reinforcement Learning

Freeway on-ramp metering is an important part of freeway traffic control,which can effectively alleviate the main line congestion of freeways.Deep reinforcement learning algorithm has the advantages of deep learning and reinforcement learning.It is an efficient intelligent algorithm that can make decisions in a continuous space and self-train by interacting with the environment.Based on deep reinforcement learning,the single on-ramp metering algorithm and coordinated ramp metering algorithm are studied.Combining the DDPG algorithm with the on-ramp metering algorithm ALINEA,a single on-ramp metering algorithm with dynamic parameter adjustment is established.The multi-agent algorithm MADDPG is introduced into coordinated ramp metering method.The multi-agent-based coordinated ramp metering is realized by the MADDPG algorithm controlling multiple entrance ramps.Macroscopic traffic flow models METANET and LWR are analyzed.According to the measured data of the road segment,the improved particle swarm optimization algorithm is used to identify the model parameters.According to the identify results,the traffic flow simulation platform is constructed based on the METANET model.The problem that the deep reinforcement learning algorithm needs the actual road segment as the training environment is solved by the platform which can replace the actual road segment as the environment for deep reinforcement learning algorithm training and testing.Based on the classical ALINEA algorithm,the Deep Deterministic Policy Gradient algorithm(DDPG)is introduced.Take the segment state information as input,the ALINEA algorithm PI parameters as output,and the dynamic parameter adjustment DDPG-ALINEA algorithm is established.In addition,ramp queue length is added into the state space and reward function,and the DDPG-ALINEA algorithm that constrained by ramp queue length is designed.The DDPG-ALINEA algorithm constrained by ramp queue length can make up for the lack of the ALINEA algorithm without considering the length of the ramp queue.Based on the DDPG algorithm,the multi-agent reinforcement learning algorithm is introduced into the coordinated ramp metering.The principle of MADDPG algorithm based on centralized training with decentralized execution framework is studied.Based on the characteristics of sharing action information between agents in MADDPG algorithm,the coordinated ramp metering algorithm is established.Based on the established traffic flow simulation platform,the DDPG-ALINEA algorithm and the MADDPG coordinated ramp metering algorithm are trained and tested.The simulation results show as follows:In the condition without the queue constraint,the DDPG-ALINEA algorithm reduces the Total Travel Time(TTT)of the controlled road segment during the congestion period by 13.93%.Compared with the ALINEA algorithm and the PI-ALINEA algorithm,the TTT of DDPG-ALINEA algorithm decreases by 1.67%and 1.51%respectively.In the condition of ramp queuing constraint,the DDPG-ALINEA algorithm has a 35.35%reduction in Total Waiting Time(TWT)and a 9.29%reduction in Total Time Spend(TTS)compared to the PI-ALINEA algorithm.Compared with the uncontrolled condition,the MADDPG coordinated ramp metering algorithm reduces the TTT of the controlled road segment by 23%.That the DDPG-ALINEA algorithm and the MADDPG coordinated ramp metering algorithm based on deep reinforcement learning enhance the control effect of the entrance ramp on the freeway traffic flow and make the freeway transportation efficiency increase is shown in the simulation results.The feasibility of the above method applied to the freeway entrance ramp control is proved.