Multi-agent Reinforcement Learning With Hierarchical Game Model For Coordinated Control Of Regional Road Network Signals

With the rapid development of accelerated urbanization process,the road traffic flow is increasing,which leads to the increasingly serious traffic congestion problems.Facing the demand to reduce the pressure on the road network and improve the efficiency of vehicle operation,the research on the coordinated control methods among the intersection signals in a urban area road network has become a current research hotspot.The traffic flow at each intersection in the urban road network has a large correlation,so introducing a reasonable coordination mechanism can effectively control the traffic signals and alleviate traffic congestion.Aiming at the shortcomings of the existing research on the coordinated control of the traffic signals in regional road networks,this paper analyzes the game relationships between important intersections and between important and secondary intersections using game theory for a multi-agent system composed of multiple intersection signals in a road network region,takes into account the traffic control strategies between and within sub-areas of the road network,and constructs the Nash-Stackelberg hierarchical game model(NSHG)and the NashFo F hierarchical game model(NFHG)are developed.On this basis,reinforcement learning algorithms based on NSHG and NFHG,i.e.,NSHG-QL and NFHG-QL algorithms,are proposed.Moreover,the deep reinforcement learning algorithms,namely NSHG-DQN and NFHG-DQN algorithms,are proposed to overcome that some agents have difficulty in converging the learning process due to the dimensional explosion problem.Finally,it is verified that the proposed algorithm can reduce the average travel time,average time loss and improve the average speed and roadway occupancy of vehicles within the intersection in the simulation network compared with the control algorithm,and is able to obtain a better joint control strategy to achieve the traffic level optimization of the roadway network.The main research contents are as follows:Firstly,in order to solve the Nash equilibrium computational complexity problem caused by multi-intersection game,this paper proposes a Nash-Stackelberg hierarchical game model which considering the importance and the game relationship of different intersections in the road network.In the upper game,the important intersections in the two sub-areas are used as the game subject at the upper layer,and the joint action strategy of the primary intersection Agent(P-Agent)is coordinated to converge to the Nash equilibrium;in the lower game,the secondary intersections are used as the lower game subject,and on the basis of satisfying the P-Agent in the region,the joint action strategy of multiple secondary intersection Agents(S-Agent)is coordinated to converge to the Stackelberg equilibrium.Similarly,a Nash-Fo F hierarchical game model is proposed to coordinate the P-Agents and S-Agents in the lower-level game for a fully cooperative or fully adversarial game using Friend-or-Foe equilibrium.Secondly,the multi-agent reinforcement learning algorithms,i.e.,NSHG-QL and NFHG-QL,are proposed based on the NSHG and NFHG models and applied to road network traffic signal control optimization.Simulation experiments show that the NSHG-QL prioritizes the control optimization of the P-Agent and enables the P-Agent to converge to the optimal joint strategy during the learning process with better control effects compared with the control algorithms that only consider a single game relationship.Although NFHG-QL can achieve the learning convergence of the P-Agent,its control optimization effect is weaker than the NSHG-QL algorithm.Finally,the NSHG-DQN and NFHG-DQN algorithms are proposed in combination with deep learning techniques to address the problem that the learning process cannot converge due to the dimensional explosion of S-Agents.Experiments show that the NSHG-DQN can significantly improve the traffic level of the road network compared with the control algorithm based on considering only one game relationship model and is better than NFHG-DQN.The Nash-Stackelberg hierarchical game model proposed in this paper can reflect the importance degree and game relationship of different intersections in the road network,solve the Nash equilibrium computational complexity problem caused by multi-intersection game,and obtain the global optimal joint strategy.