Research And Application Of Multi-Agent Reinforcement Learning In Traffic Signal Control

Urban traffic congestion has a serious impact on the global economy and environment.Improving the efficiency of urban traffic signal control is one of the relatively “efficient and low-cost”ways to alleviate urban traffic congestion.Because of the complexity and uncertainty of the urban traffic environment,the mechanism of its internal operation cannot be accurately modeled by mathematical analysis.Thus,the urban traffic signal control is suitable for the reinforcement learning(RL)method,which is“data-driven,self-learning,model-free”to compute and control.Researchers have conducted extensive research work on urban traffic signal control using RL and deep reinforcement learning(DRL)models.In recent years,the study of local traffic signal control has evolved from local optimization at a single intersection to global optimization at multiple intersections.The Multi-Agent Reinforcement Learning(MARL)approach has become one of the major research models.It has achieved good results in existing urban traffic simulation environments.However,the current research still has the following shortcomings:(1)the current traffic simulation environment can not simulate the real traffic environment closer to the real traffic environment,resulting in a particular gap between the effect of urban traffic signal simulation control and the actual road test;(2)the control effect of the MARL algorithms for municipal multi-road traffic signal control did not obtain good results,which still has room for improvement by the combination of game theory and communication theory.The MARL methods are not studied deeply enough;(3)the industrial deployment of the urban traffic signal control algorithm at multiple intersections is not studied enough.In response to the above problems,this paper has carried out a meticulous research and achieved the following results:(1)Established an urban multi-road traffic flow time series text dataset(Multi-Road Traffic Dataset,MTD),built an urban simulation environment(Urban Simulation Environment,USE)closer to the real road conditions.(2)Promoted the deep integration of MARL theories,algorithms,and intelligent transportation applications to expand the use of MARL by combining the Nash Equilibrium theory and communication theory.Proposed a Nash Equilibrium based algorithm Nash-Advantage Actor-Critic(Nash-A2C),the Nash-Asynchronous Advantage Actor-Critic algorithm(Nash-A3C),and automatic communication of multiple agents learning algorithm(MAAC).(3)In view of the insufficient research on the industrial-level deployment of the MARL algorithm at multiple intersections in the city,proposed the Mixed-MARL algorithm based on a hybrid computing architecture of cloud computing,fog computing,and edge computing.(4)Integrated urban simulation environment,hybrid computing architecture,MARL urban traffic signal control algorithm,and proposed a comprehensive application computing framework(General City Traffic Computing System,GCTCS)for urban traffic signal control,and combine Nash equilibrium,multi-agent communication,and hybrid Computational architecture advantages,proposed a combined algorithm General-MARL.In this paper,The urban simulation environment USE constructed uses the method of dynamic traffic flow prediction to achieve an effect that is closer to the real situation than the traditional simulation environment;The experimental results of proposed MARL optimization algorithms(Nash-A2 C,Nash-A3 C,MAAC,Mixed-MARL,General-MARL)in the simulation environment USE are better than traditional baseline algorithms.The General-MARL algorithm,which combines Nash equilibrium,multiagent communication,and hybrid computing architecture,has achieved a certain degree of results in alleviating congestion at multiple intersections and reducing network delays(exceeding the baseline method by 23.2%,communication delay Decrease by 11.7%).