Research On UAV Clusters Communication Topology Control Based On Deep Reinforcement Learning

UAV clustering has increasingly become the main hot research topic in the development and application of UAV systems.As the basis of data transmission and collaborative interaction between UAVs,network connectivity within the cluster has received widespread attention.The flying ad hoc network is the application of the selforganizing network in the UAV system.It has the characteristics of temporary networking,rapid deployment,no control center,and dynamic change of the topology.The research on the cluster communication network in this paper is based on the flying ad hoc network.This paper focuses on the problem of drone cluster network topology control,introduces deep reinforcement learning algorithms on the basis of traditional models,and ensures stable point-to-point effective connectivity between drones through topological single-connection and bi-connection control.The main research contents are summarized as follows:The establishment of cluster network topology control model.Firstly,the cluster network topology model is established based on graph theory and the characteristics of ad hoc networks in flight,and the related definitions of network topology are given.Then the classical cluster mobility model is simply introduced.combined with the cluster mobility theory,the cluster mobility control model based on network connectivity is established,which provides model support for the follow-up control research on network connectivity.Cluster network single-connection topology control.Aiming at the problem of rebuilding the connected network when the cluster communication network is interrupted,a node movement control method based on multi-agent deep reinforcement learning is proposed.Firstly,aiming at the distributed topology control problem,the training environment is established,the reward function is designed,and the multi-agent deep reinforcement learning algorithm is introduced to train the multi-agent to learn the strategy of constructing the single connected network.On this basis,the reward function is decoupled and the algorithm framework is improved and the reward decoupling multi-agent deep reinforcement learning algorithm is proposed.Finally,the effectiveness of the improved algorithm and the superiority of the strategy are proved through the experimental simulation.Cluster network bi-connection topology control.In order to solve the problems of poor robustness and easy paralysis of single connected cluster networks,a topology control algorithm based on deep reinforcement learning was proposed to construct biconnected networks.First,base on that graph theory,the problem of constructing a twoconnected network is transform into the problem of eliminating key points in the topology graph,and the inclusion relation between k-jump key points and global key points is proved,so that the connectivity of the network topology is judged under the distributed condition.Then,combined with the cluster mobility model,a node mobility control algorithm based on virtual force field is proposed.To optimize the topology biconnected control from the perspective of strategy,the reinforcement learning control based on virtual potential field is proposed.Finally,the simulation results verify the superiority of the proposed two algorithms compared with the traditional algorithm.Aiming at the connectivity of UAV cluster communication network,this paper proposes a distributed control algorithm based on deep reinforcement learning by referring to the classical cluster movement theory and combining with the perceptual decision-making ability of deep reinforcement learning.The single connected control algorithm is realized based on the framework of "centralized training and distributed execution" in multi-agent reinforcement learning.The bi-connected control algorithm is based on graph theory knowledge to transform the global connected problem into the local key point elimination problem.Compared with the traditional cluster mobility algorithm,the two algorithms have better performance.