The Intelligent On-ramp Metering At Urban Expressway Weave Area Based On Reinforcement Learning

Compared with freeway ramp system,the on-ramp and off-ramp of expressway ramp system is closer,and the weave area is more prone to crowded and spreading to the upstream and on-ramps.Ramp metering is used to solve the mainstream congestion and the on-ramp queue,Ramp metering is usually used to solve the mainstream congestion and the on-ramp queue problem.Traditional method has defects,such as traffic model construction,model parameter calibration,dependenting on a priori knowledge and control hysteresis.Based on the reinforcement learning,this paper puts forward the intelligent ramp metering.This method directly uses the actual road segment detector to collect data to coordinate multiple on-ramps metering on the expressway network,and the control effect is less affected by the control parameter setting,which is a kind of modelless,self-learning intelligent ramp metering.This paper defines the basic concept of urban expressway interweaving area,and analyzes the traffic characteristics based on the actual collected data,which provides the basis for the construction of the simulation road network.Introduce the basic concepts of reinforcement learning,analyze the merits of Q learning algorithm and its improved algorithm SARSA.Based on the SARSA algorithm,a single ramp metering model is established.Chose metering rate as behavior,and traffic capacity of the expressway system as rewad,the traffic volume,occupancy rate of the expressway system and queue length as state.Determining the behavior space,the state space,the reward funciton,and finding behavior selection mechanism.Formating local ramp metering based on SARSA(SRM).Based on the multi-agent reinforcement learning,the local intelligent ramp metering is extended to the coordinated metering to control multiple on-ramps.Based on the cooperative graph method,the Q-value matrix update rule,the joint reward function and the joint optimal behavior strategy are given to form the coordinated ramp metering based on SARSA control method(CSRM).Correcting the reward function,considering the ramp queue length limit and the critical occupancy rate of the weave area,to make the algorithm make full use of the idle queuing space,and increase the traffic volume of the weave area.The Vissim-Matlab simulation platform is constructed by Matlab and Vissim microscopic simulation software.The local and coordinated ramp metering of this paper are compared with the traditional methods.Compared with the traditional method ALINEA,the average delay of SRM has decreased by 16.10%,the total delay is reduced by 12.10%,the average speed is increased by 6.77%,and the total travel time is reduced by 3.78%.In addition,the SRM's ramp queue length is less than ALINEA.During the peak traffic congestion,the average traffic volume on the weave area of the crowded ramp is increased 210 veh/h compared with ALINEA.Compared with BOTTLENECK,the average delay of the CSRM is 4.31%;the total delay is reduced by 2.09%,the average speed is increased by 3.69%,and the total travel time is basically the same.In addition,the CSRR's idle ramp queue length is longer than BOTTLENECK.The queuing length of crowded ramp is less than t BOTTLENECK.During the peak traffic congestion,the average traffic volume on the weave area of the crowded ramp is increased 140 veh/h compared with BOTTLENECKIn this paper,the application of intelligent ramp metering in expressway traffic field is discussed,which provides a new idea for the following research.