UAV-assisted Communication Deployment Strategy Based On Reinforcement Learning

With the continuous growth of user needs and the ever-changing network environment,UAVs equipped with relay equipment have been widely used in military,agriculture,urban surveys,disaster reconstruction,etc.On the one hand,the relatively high flight height of the UAV has a large line-of-sight probability for ground users,which greatly improves the network coverage performance.On the other hand,drones communicate and share information with each other in the air,which is more flexible and easier to adapt to different environments.As a relay auxiliary ground base station,the UAV expands the coverage and communication range,provides data access for more ground terminals,and meets the service needs of more users.The deployment of a single UAV base station is limited by power consumption,bandwidth resources,coverage,etc.,and the scenarios in which multi-UAV joint networking can be applied are more extensive,but the complexity of the ground environment brings practical challenges to UAV deployment.Current deployment efforts focus on different research content,but their assumptions and constraints also have different shortcomings.1)Most of them take two-dimensional deployment at a fixed height,rather than looking for the best deployment location in three-dimensional space;2)In order to find the optimal joint deployment of the UAV,the location information or distribution of the user is required;3)Assuming entities with global information,implement centralized deployment;4)The connectivity of the communication network between the UAV and the base station is not considered;5)The limited power of the drone and the amount of electricity consumed by the drone to fly to the target position are not taken into account.Therefore,the main contributions of this paper are as follows:（1）According to the network architecture of UAV base stations combined with ground base stations,we propose a hybrid deployment problem which UAV assist ground base station to communication.Find the best deployment location of the UAV in three-dimensional space,so that the UAV and the ground base station are within their respective communication ranges（the communication range is determined according to the Air-to-Ground channel model）,maximizing the number of mobile users that can be served globally,and ensuring the connectivity of the UAV to the base station.（2）A multi-UAV distributed reinforcement learning method SDQ-H（Semi Distributed Q learning-Height）is proposed to achieve three-dimensional deployment of distributed UAV under the condition that the location of mobile users is unknown.In order to ensure the connectivity between users and ground base stations after deployment,and to solve the problem of unknown user location information,a hierarchical reward mechanism and communication strategies between UAV base stations and between UAV base stations and ground base stations are proposed.At the same time,in order to extend the service time of multiUAV auxiliary communication network,after the completion of SDQ-H deployment,this paper proposes a centralized deployment algorithm SDQ-H-RL（Semi Distributed Q learningHeight-Reinforcement Learning）,including multi-UAV three-dimensional position optimization scheme and the principle of close-range deployment,which effectively improves the total number of user service hours and maximizes network service time.（3）Experimental evaluation is carried out on the above two methods in four different user distributions（Tokyo,New York City,Cluster,PPP）,combined with the communication channel model between UAVs,base stations,and ground users.In the SDQ-H method,the optimal parameters are determined by a large number of premise experiments,and then compared with the existing heuristic and centralized deployment algorithms,the method is experimentally verified on the influencing factors such as environmental parameters,SINR connection threshold,number of ground base stations,user density and communication distance,and the results show that SDQ-H has good superiority in coverage performance.In the SDQ-H-RL method,this paper sets the two evaluation indicators of the cumulative total service time slots and the network lifetime of the user,and compares them with the improved extremum optimization algorithm on different factors such as the initial distribution of UAVs,the distribution of ground base stations and the user density,and the experimental results show that the method can effectively extend the network lifetime and improve the total number of user service time slots.