Deep Reinforcement Learning For Resource Management Research In Cell-Free Networks

Cell-free network is considered as a promising architecture for satisfying more demands of future wireless telecommunication networks,where multiple distributed access points(APs)coordinate with an edge cloud processor to jointly provide service to a smaller number of user equipment(UE).Cell-free network ignores the restriction of cell boundaries.For the utmost of the telecommunication and compute resource and the remission of cell-edge coverage problems,cell-free network empowers all APs full connected with all UEs.For eliminating the problems such as inter-user interference,non-orthogonal pilot contamination and excessive control signaling in cell-free network,an intelligent and efficient design for beamforming is highly desired in uplink and downlink signal transmission.Furthermore,the line-of-sight(LoS)links between APs and UEs are frequently blocked by high-rise buildings with high probability,which leads poor reliability and energy efficiency in cell-free network.In order to mitigate the signal interruption problems,an emerging material called reconfigurable intelligent surface(RIS)is widely employed to reconfigure the wireless propagation environment by formulating virtual LoS links between APs and UEs.Benefited from the intelligent deployment and design for the phase shift and position selection,RIS enables to create the reflected channels between APs and UEs in an adjustable way,and realizes the enhancement of system performance in various practical scenarios.In terms of dynamic and large number of accesses in practical wireless networks,deep reinforcement learning(DRL)is widely employed to handle the dynamic problems with large scale state and action space by the approximate representation of neural network,in order to obtain the intelligent and efficient beamforming and RIS joint design.Deep deterministic policy gradient(DDPG)algorithm is a kind of strategy learning method for continuous behavior in DRL.Therefore,this paper utilizes DDPG algorithm to optimize the beamforming and RIS joint design.The main work and research of the thesis are as follows:(1)The design for beamforming is investigated for maximizing the long-term energy efficiency with the aid of DDPG algorithm in cell-free network.Firstly,based on the minimum mean square error channel estimation and exploiting successive interference cancellation for signal detection,the formulation of energy efficiency is derived,which is a function of beamforming design.Secondly,a DDPG-based beamforming design is proposed for the optimization of energy efficiency,and our proposed beamforming design is capable of converging to the optimal energy efficiency performance based on the simulation results.Finally,the influence of hyper-parameters on the energy efficiency performance is further investigated,and it is demonstrated that an appropriate discount factor and hidden layer size enable to facilitate the energy efficiency performance.(2)The DDPG-based beamforming and RIS joint design is proposed for energy efficiency optimization in RIS aided cell-free network.The design jointly optimizes beamforming design,phase shift and position selection of RIS,which alleviates the signal interruption caused by highrise buildings and maximizes the energy efficiency in cell-free network.Aiming at the multivariable optimization,this paper proposes two joint designs based on different network structures in DDPG algorithm,and verifies the optimal network structure setting scheme to deal with multivariable joint optimization problem by simulation comparison.Finally,the impact of the signal transmission power,the number of reflection elements in multiple RISs on energy efficiency performance is further studied,and it is concluded that the appropriate network setting elevating the system performance in RIS aided cell-free network.