Research Of Deep Reinforce Learning Based Resource Allocation In Cellular Heterogeneous Networks

In the future,6G will develop towards the Internet of everything.With the rapid growth of mobile devices and Internet of Things devices,as well as the surging demand for mobile data traffic,many problems gradually emerge such as spectrum resource depletion and insufficient system capacity.In order to meet the requirements of wireless communication,heterogeneous networking technology can effectively increase the system capacity by increasing the number of multi-type small base stations and shortening the distance between terminal devices and base stations.At the same time,heterogeneous network can solve the problem of spectrum shortage by sharing the same channel with macro station to achieve high spectral efficiency.However,in heterogeneous networks,due to the coexistence of a large number of micro-base stations and a large number of terminal devices,the distance between the terminal devices and the micro-base stations is closer and the interference is more serious,which greatly affects the user Quality of Service(QoS).Therefore,the correct choice of channel and base station will make efficient use of spectrum,data traffic and other resources,which can greatly alleviate the problem of interference and insufficient system capacity.Therefore,how to realize the joint optimization of user associated base station and channel allocation on the premise of guaranteeing the QoS of end users has become an urgent problem to be solved.Traditional optimization algorithms such as game theory and convex optimization need almost complete channel state information and base station information to optimize joint optimization problem.And traditional algorithms are mostly based on the current state of network without thinking of the future system benefits from the network.In addition,in the process of joint optimization of the user association and channel allocation,the number of base stations and users is great in heterogeneous network,which lead to the increase in the state space and motion space.It will cause huge amount of calculation in resources allocation.To solve the above problems,a distributed optimization method of multi-agent deep reinforcement learning Dueling DQN(MADDQN)algorithm was proposed.The multi-agent DQN method can solve the difficulty of obtaining complete Channel State Information(CSI)by constantly interacting with the environment to obtain information.Meanwhile,reinforcement learning can consider the benefits of the system from a long-term perspective by constantly discounting past benefits.To solve the problem that DQN convergence is not fast enough,Dueling architecture is introduced.By dividing Q into mean value and advantage value,system information can be extracted quickly to complete the convergence of the strategy.However,the above MADDQN algorithm also has the problem of poor performance of sum rate(system capacity)in policy convergence,which is caused by insufficient information extraction.To solve this problem,a Double DQN algorithm(MAPD3QN)based on Multi-agent Prioritized Experience Replay(PER)and Dueling architecture is proposed.PER technology makes efficient use of important but rare empirical data by classifying the importance of empirical data.Double DQN monitors the update of Q value by adding target network,so that the information extracted by deep neural network can be correctly estimated and better strategies can be found.Simulation results show that the MADDQN and MAPD3 QN algorithms proposed in this thesis can achieve fast convergence of user associated base station and channel allocation strategy while ensuring user QoS,and achieve good sum rate(system capacity).