Research On Deep Reinforcement Learning-based Active Traffic Flow Control Strategies At Distant Downstream Bottlenecks Of Expressway

With the rapid increase of the demand for the vehicle,the traffic congestion has become more and more serious on expressways.Existing active traffic flow control strategies,including variable speed limit control and ramp metering control,are the most effective control strategies to alleviate congestion on expressways.The ramp metering control regulates the flow of the ramp into the main road,and the variable speed limit control dynamically adjusts the speed limit of the upstream section of the bottleneck.Both control strategies are designed to increase the flow by preventing the capacity drop of the bottleneck section.In recent years,there have been many researches on the bottleneck sections of the expressway,but there are few studies on the further downstream bottlenecks of the expressway.In many practical cases,bottlenecks with smaller capacities than the merging area may exist further downstream,due to the phenomenon of downhill slopes,bends,tunnels,bridges,and reduction of lanes in the downstream,etc.When the distance is far,the delay effect of the system will lead to the substantial fluctuations in the flow rate of main line and ramp metering,which will result in a serious threat to the efficiency of traffic and road safety.In this case,on one hand,the controller cannot only consider the traffic flow information at the bottleneck,and how to make full use of the traffic flow information at the upstream and downstream of the section is the key problem.On the other hand,the controller needs to overcome the oscillation under the condition of long control distance,and how the control algorithm adapts to the arbitrary change of distance between control point and downstream bottleneck is also an important problem.First,based on the Deep Q Network(DQN)algorithm,a variable speed limit control strategy and a ramp control strategy are proposed in this paper.The main three parts of the algorithm are defined,which includes the input state,the output action and the reward value.Introduce the training process of DQN-based ramp metering control strategy and DQN-based variable speed limit control strategy.After effective training,the agent can take a series of actions to maximize the reward.Secondly,the study develops the traffic simulation model for active control,and the cellular transmission model(CTM)is modified and improved.Build the CTM simulation platform and Python development environment,real-time interaction between platforms is realized through modular design,and the simulation calculation process for multi real-time control strategies is proposed.Then the effect of DQN-based control strategy is analyzed.The difference between feedback algorithm and Q learning based algorithm in control mechanism and control effect is compared.Finally,the results show that the proposed DQN algorithm is superior to other algorithms in ramp metering control and variable speed limit control.In ramp metering control,the DQN algorithm reduces the total travel time by 34.09% to 35.39% in the stable demand scenario,and the DQN algorithm reduces the total travel time by 40.38% to 41.29% in the fluctuating demand scenario.In variable speed limit control,the DQN algorithm reduces the total travel time by36.04% to 37.77% in the stable demand scenario,and the DQN algorithm reduces the total travel time by 44.51% to 45.65% in the fluctuating demand scenario.The results also indicate that the DQN algorithm improves the capacity of the bottleneck area and reduces the spatialtemporal distribution of the congestion state,so it can minimize the total time spent.The research results also show that the DQN based control strategy has the advantages of strong predictive ability,fast convergence speed,high action accuracy,etc.The proposed algorithm can still get better control effect in the system with time delay.