Mechanism Of Knowledge Reusing For Multi-agent Reinforcement Learning

As one of the research hotspots in the field of artificial intelligence,multi-agent technology has a wide range of applications in UAV cluster,missile system,resource scheduling and other fields,and also has an important application prospect in national defense and military modernization.In recent years,thanks to the development of deep reinforcement learning technology,the multi-agent deep reinforcement learning method arises at the historic moment,compared with the traditional method,it showed obvious advantages in helping multi-agent cope with open scenes and dynamic change of scenes,thus in the field of multi-agent plays a pivotal role.However,the multi-agent reinforcement learning method still has the following problems in practical application: the trained strategies often have strong task relevance,and most task strategies can only target specific task scenarios,when various elements in the task scene(such as the number of agents,environmental elements,etc.)change,the effect of the previously trained task strategy often decreases significantly,or even completely fails.It is necessary to retrain the task strategy according to the new task environment.However,multi-agent reinforcement learning method is a kind of ”data-driven” method,which requires multi-agents to find the best strategy through continuous trial and error in the task.Training a new task strategy often requires a large amount of data and a long time of training,which has the problem of”large amount of training data and long training time”.Aiming at the problem of ”large amount of training data and long training time”of multi-agent reinforcement learning method,from the perspective of knowledge reuse,this paper makes the agent quickly adapt to the new task,improve the task training effect and shorten the training time by reusing the knowledge in the past task strategy or the knowledge between agents.The main work focuses on knowledge reuse across task and knowledge reuse intra task of multi-agent,and proposes the following three methods:(1)A multi-agent cross task knowledge reuse method based on knowledge distillationAiming at the problem that multi-agents are difficult to overcome the environmental instability in new tasks,a multi-agent cross task knowledge reuse method based on knowledge distillation is proposed.The method makes use of previous policy model of multi-agent task as the historical experience model,in the process of target agent model training,the target model not only needs to learn knowledge according to the feedback of the task environment,but also needs to reuse knowledge and adjust the action guidelines from the historical experience model in the current task environment behavior.This method can successfully shorten the training time of multi-agent in both various scenarios including cooperation scenarios and confrontation scenarios.(2)A multi-agent cross task knowledge reuse method based on cognition transferTo solve the problem that it is difficult for agents to model the task environment in a new task,a multi-agent cross task knowledge reuse method based on cognitive transfer was proposed.Cognition in multi-agent system refers to the agent’s knowledge about what is a good action in the task environment and how to evaluate the good or bad action.The method makes use of the thought that agents should have similar cognitive in similar tasks,retention and reuse cognitive model that is set up according to the task environment in previous tasks,to guide the current task agent policy model in the study.The policy model not only in the past the cognitive model guidance at the same time,also according to the current environment to optimize their own strategies,thus realizing the transfer of agent cognition between similar tasks.This method can help agents adapt to the task environment quickly in various scenarios including cooperation scenarios and confrontation scenarios.(3)A multi-agent intra task knowledge reuse method based on cognition consistencyIn order to solve the problem of poor cooperative performance due to different learning rates of agents in multi-agent tasks,a knowledge reuse method based on cognitive consistency was proposed.This method uses the idea of mutual learning for reference,and makes use of the feature that homogeneous agents should maintain cognitive consistency in a task,so as to help agents to reuse the knowledge learned by other homogeneous agents in tasks while training and accumulating knowledge,and promote multi-agents to reach agreement and complete better cooperation.The cognitive consistency in multiagent system means that homogeneous agents have the same knowledge about what is a good action in the task environment and how to evaluate the good or bad of the action.This method can help multi-agent to cooperate with each other quickly and shorten training time in various cooperative scenarios.In this paper,the effectiveness and robustness of the above methods are verified in cooperative and adversarial simulation environments.Experimental results show that the above methods can effectively help agents to learn quickly,reduce the time required for training convergence,and accelerate the generation of agents’ strategies.