| In a multi-agent environment,the continuous change of each agent’s strategy will lead to the non-stationarity of the multi-agent environment,so the reinforcement learning problem in a multi-agent environment is usually modeled as a decentralized partially observable Markov decision Process(Decentralized Partially Observable Markov Decision Process,Dec-POMDP),which brings challenges to the cooperation between agents.In order to overcome and alleviate the non-stationarity of the multi-agent environment,the mainstream method is to adopt the framework of Centralized Training Decentralized Execution(CTDE).This thesis is based on the framework of CTDE,and studies the cooperative decision-making of multi-agent based on the Multi-Agent Proximal Policy Optimization(MAPPO)algorithm for multi-agent proximal policy optimization.(1)In order to alleviate the non-stationarity of the multi-agent environment,a multiagent communication mechanism based on weight scheduling and attention module is introduced.Different agents can alleviate the non-stationarity caused by local observations through information exchange between agents,assisting in the collaborative decisionmaking of agents.The specific method is to introduce a communication module in the policy network part.The communication module is composed of a weight generator,a weight scheduler,a message encoder,a message pool and an attention module.Among them,the weight generator and weight scheduler will generate weights as the selection basis for communication,the message encoder is used to compress and encode communication information,the message pool is used to store communication messages,and the attention module realizes the interactive processing of the agent’s own information and communication information.(2)In the CTDE framework,global information is introduced during centralized training to alleviate environmental non-stationarity.In the MAPPO algorithm,the input of the centralized value network contains global information,and the processing of global information has an impact on the estimation of the value function.This thesis proposes a global information processing module based on the attention mechanism and deep and shallow feature processing.The global information and the local observation information of each agent are input into the attention module to obtain the simplified feature information,after deep and shallow layer feature processing,it is used as the input of the value network.Combining the above improvements,this thesis proposes a Multi-Agent Communication and Global Information Optimization Proximal Policy Optimization(MCGOPPO)algorithm,and conducted experiments in the Star Craft Multi-Agent Challenge(SMAC)and the Multi-Agent Particle Environment(MPE).The experimental results show that the improvement has achieved certain effects,which can better alleviate the non-stationarity of the multi-agent environment,and improve the collaborative decision-making ability among the agents. |
