Urban Road Network Traffic Control Method Based On Game Theory And Multi-agent Reinforcement Learning

The controlled process of traffic flow at signalized intersections has an important influence on the performance of urban road network.However,the current optimizations of urban road network traffic control schemes are still in a circular dilemma,which is in order of optimization,performance improvement,demand increase,performance deterioration and re-optimization.The dilemma is faced with the problem of cycle shorted,under section of accelerating traffic demand change and enhancing traffic flow dynamics caused by the rapid development of economy,technology and urban construction.Optimizing traffic control schemes frequently will increase the cost of urban traffic construction.Inability to restrain stochastic demand based traffic congestion is the main problem faced by existing traffic control methods.Therefore,it is necessary to research the urban road network control method for the dynamics and stochasticity of traffic flow in urban road network.The purpose and significance of the research in this paper on aspect of urban road network traffic control methods are as follows:(1)further supplement and improve the theoretical system of urban road network traffic signal control field between the distributed traffic control methods and the coupled methods of traffic assignment and signal control;(2)to make traffic signal control system has the self-adaptive improvement ability,which can co-evolve with the topology structure and the traffic demand in urban road network,so as to avoid the circular dilemma of iterative optimization.In this paper,the research is carried out from three aspects: the distributed dynamic traffic assignment for stochastic travel demand;the distributed traffic signal control to deal with the impact of stochastic travel demand;and the traffic signal control system to deal with derived impact from the damaged network nodes.(1)The distributed dynamic traffic assignment method,which can assignment stochastic travel demand and restrain traffic congestion from the source,is the key basis for the following research.The multi-agent groups with heterogeneous advisers were constructed,in which heterogeneous advisers' s suggestions are used to restrict the action space of decision-makers.In the limited action space,the form of mixed strategies improves the learning efficiency of multi-agent reinforcement learning in dynamic traffic assignment task.The differentiated reward function mechanisms were constructed to enable agents to realize user equilibrium principle.The adaptive learning rate mechanism was designed to improve the sensitivity of the method to random travel demand and traffic state changes and to enhance itself re-learning ability.The empirical analysis shown that the distributed dynamic assignment method can effectively improve the overall throughput level of the network and reduce the average travel time of individual travelers in the urban road network,in which the travel demand distribution results complied with user equilibrium principle.Additionally,the study also shown that coupling with the distributed traffic signal control method can effectively reduce the level of travel delay in the road network.(2)The distributed traffic signal control method is the important core of the research in this paper,which was created to maintain the stability of urban road network performance from the perspective of responding to the stochastic impact of travel demand,so as to restrain and alleviate the occurrence and spread of traffic congestion.The concept of mixed strategy Nash equilibrium in the game theory was introduced to improve the decision-making process of multi-agent reinforcement learning,so that agents can obtain global information implicitly,and enhance theirself ability to maintain the stable performance of road network under different stochastic travel demand states.Based on the Nash equilibrium solution of mixed strategy,Jensen-Shannon divergence was introduced to construct an adaptive learning rate mechanism,which enhances the sensitivity of signal control agents to local traffic flow state changes,and keeps agents with ability to restart learning after convergence.The empirical analysis shown that the distributed traffic signal control method had a good control effect in dealing with the sudden increase of travel demand,the sudden change of travel demand distribution between the origin-destination,the unequilibrium travel demand distribution in the road network,and the randomness influence caused by stochastic arrival rate,which can maintain the travel delay in the urban road network at a relatively low level.However,the verification also shows that this method is only suitable for urban road network with medium traffic input.It means that the method presented in this paper is a deep excavation on the aspect urban road network temporal and spatial resources.(3)The enhancement of signal control robustness in response to network node failure focuses on enhancing the ability of signal control methods to maintain the performance of the network when the network structure is damaged,which is a supplement to the important core.Based on the multi-agent system,a method of measuring the node level in urban road network was constructed to realize the identification of key nodes of road network.By introducing the traffic state of the nodes in the road network,the anisotropic influence propagation mechanism of the nodes was constructed to realize the dynamic evolution of node interaction.According to the difference of node interaction structure,the corresponding signal control agent mixed strategy Nash equilibrium solution decision process and reward function mechanism were modified.The empirical analysis shown that in the case of a small number of node failures,the method suggested in this paper can maintain the travel delay of urban road network at a relatively low level,and effectively restrain the decline speed of network performance when the invalid time of node increase.However,when the topology of the road network is seriously damaged and the carrying capacity cannot meet the travel demand,the robustness enhancement mechanism cannot improve the performance of the signal control method.The major contribution of this study can be summarized as follows:(1)the dynamic traffic assignment method constructed in this paper can inhibit the formation of traffic congestion from the root,and the method can independently perform the task of traffic assignment.Coupled with distributed control,the method can effectively restrain congestion,and it is also the key interface of the signal control robustness enhancement mechanism;(2)the distributed traffic signal control method for the influence of stochastic travel demand can perceive the global information implicitly in the local interaction process,and effectively alleviate and restrain the traffic congestion induced by random travel demand.(3)the signal control robustness enhancement mechanism builds the interaction between nodes and improves the failure robustness of network nodes for the signal control method.Through the integration of distributed dynamic traffic assignment,distributed traffic signal control and signal control robustness enhancement mechanism,the traffic control system can effectively cope with frequent optimization dilemma and has the ability to co-evolve with the urban road network.