| Ultra-dense network(UDN)is considered as one of the key technologies for the explosive growth of mobile traffic demand on the Internet of Things.It enhances network capacity by deploying small base stations in large quantities and it also simultaneously causes great energy consumption.Therefore,it is necessary to improve the system energy efficiency and reduce the energy cost of small base stations(SBSs)on the basis of solving the user service quality effectively.In recent years,scholars at home and abroad have devoted themselves to researching resource allocation schemes of SBSs for energy collection in cellular networks.SBSs can obtain renewable energy sources such as solar and wind energy from the natural environment through energy harvesting(EH)technology to reduce the energy consumption of traditional grids.However,there are a large number of SBSs and the channel status between the user and its served base station is more complicated in UDN scenario.So from the perspective of green communication,how to adopt an effective resource allocation scheme is a problem worth studying.In addition,Reinforcement Learning(RL),as a new optimization theory emerging in recent years,has advantages over traditional optimization methods in exploring environment,machine learning,and intelligent decision-making.Introducing RL methods into UDN resource allocation has research value.The research of this paper is how to improve the total throughput of the UDN system under the energy collection and cooperation among SBSs.The main work can be divided into two parts:1.We have used EH and energy cooperation technologies to improve the throughput of UDN system.In this problem,the main source of energy for each SBS is the renewable energy collected by itself and the renewable energy shared by other SBSs.Considering that the energy arrival process and channel information are not available apriori,we propose an optimal deep reinforcement learning(DRL)algorithm-DQN(Deep Q Network)and DDPG(Deep Deterministic Policy Gradient).which can dynamically determine the power allocation of each SBS according to the battery power,the number of served users and the channel status over finite horizon.The research shows that the power allocation scheme based on DQN needs to quantify the action space(base station power allocation)and divide it into fixed actions to select,which will lead to the sub-optimal power selection of SBSs.The DDPG-based scheme can select a suitable action on the continuous action space to avoid the problem that the system throughput is not optimal due to quantization errors.Simulation results show that the two power allocation algorithms can converge well,and the system throughput based on the DDPG power allocation strategy can be maximized.2.Aiming at the explosion of state and action dimensions brought by the large number of SBSs in UDN,we propose a multi-agent deep reinforcement learning algorithm MADDPG(multi-agent deep deterministic policy gradient)in combination with the actual network architecture.The MADDPG-based algorithm uses centralized training and decentralized execution to treat each SBS as an independent agent with decision-making capabilities for optimal strategy training.Compared with the DDPGbased algorithm,this algorithm can reduce the neural network input information dimension effectively during the training process.Each SBS can make optimal power allocation in its own observable environment,which not only solves the aforementioned problems,but also improves the anti-interference ability of the UDN system.Finally,compared with some traditional optimization algorithms in simulation,the experimental results proved the superiority of our proposed scheme. |
