Distributed Coordinated Learning Based On Graph Model

Multi-Agent System(MAS)is an important branch of Distributed Artificial Intelligence(DAI).An MAS consists of a group of agents,and its goal is to decompose the large and complex systems into small interacted and coordinated systems,in order to achieve conflict elimination and coordination between individual agents.The distributed control method has the advantages of individual local interaction,low cost and strong robustness.Therefore MAS achieves coordination by using the distributed control method.On the other hand,the graph model can represent complex dependencies in MAS succinctly and effectively.Each node represents an individual agent in the system.The graph model can effectively characterize local state observations,behavior selection and reward reception(the goal of the system is to maximize the sum of the rewards over all nodes),so the graph-based distributed control method has important significance in MAS research.Learning from trial-and-error interactions is one of the main ways in which agents interact with the environment.Combining representation ability of the graph model with the local learning interaction of individual agent,this thesis mainly studies the distributed coordinated learning based on graph model,so as to achieve optimal or near optimal system function with minimal computational cost.This thesis mainly includes the following two research contents:This thesis first studies the multi-agent coordinated learning algorithm based on Distributed Value Function(DVF).The algorithm allows each agent to send its own value function(an estimate of a weighted sum future reward for all the nodes)to its neighbor nodes,thus takes into account the impact of not only the direct neighbor node on the choice of agent behavior,but also the other indirect effect of the node on the agent.The different position in the system and vairous performance in the learning process of each node will have different effect on the neighbor nodes.A DVF method with adaptive weight function is proposed and compared with the existing distributed reinforcement learning method in the voltage distribution problem in power grid system.Results verify the effectiveness of this method.The second work of this thesis is to study the distributed coordinated learning method by payoff propagation.The method combines the sparse cooperative learning method with the message passing algorithm in the coordination graph,and decomposes the global value function into the linear combination of the local value function by using the corresponding coordination graph.Furthermore,this method calculates the optimal joint action by the variable elimination algorithm or the Max-Plus algorithm,thus can find the optimal policy or approximate optimal policy which maximizes the overall reward of the system.This thesis realizes the distributed coordinated learning method by payoff propagation and applies the method to the single state problem and distributed sensor network problem,so as to verifie the effectiveness of the algorithm.