Research On Cooperative Caching Method Based On Federated Reinforcement Learning In Fog-Rans

With the proliferation of various terminal devices and mobile applications,the data traffic in the wireless networks shows an exponential growth.It also puts more and more load pressure on the mobile communications networks.In fog radio access networks(F-RANs),fog access points(F-APs)can store contents in advance and provide services directly to users.F-APs are closer to users,so F-APs can reduce the content request delay of users and mobile networks traffic pressure effectively.However,the caching capacity of single F-AP is limited,so cooperative caching among multiple F-APs has become a research key in order to fully utilize the caching resources and improve the quality of experience.Meanwhile,learning algorithms have become common tools for people to solve problems that are difficult to solve by traditional optimization methods.Therefore,learning algorithms are adopted to study cooperative caching methods in F-RANs.First,a cooperative caching method based on deep reinforcement learning(DRL)is investigated with limited caching capacity of single F-AP.The network model and the delay model for cooperative caching scenarios are constructed to describe the dynamic network environment and the time-varying user content request.Based on Markov decision process(MDP),the content caching replacement process for single FAP is modeled.Then,the cache replacement model for single F-AP is trained by using dueling deep Q network(Dueling DQN).The trained reinforcement learning agent can guide the F-AP to respond to the content requests from users and complete its own caching content updates.Simulation results show that the proposed Dueling DQN-based cooperative caching method can adaptively learn the dynamic network environment and the time-varying user content request.The proposed method can achieve a higher cache hit rate and a lower content request delay than traditional caching policies.Second,a cooperative caching method based on federated reinforcement learning(FRL)is investigated in the caching scenario of multiple F-APs.Considering the limited computing and caching capacity of single F-AP,the powerful cloud server and the edge layer with multiple F-APs are combined to deploy a federated learning(FL)framework.The cloud server distributes the initial DRL model to each F-AP,and each F-AP trains the model by using its own dataset.Then,the cloud server aggregates all the trained local models to obtain a global model.Subsequently,the cloud server redistributes the global model to each F-AP.The process is repeated until the global model converges.Simulation results show that compared with the method of training DRL model independently,the proposed method enables cooperative training of multiple F-APs,speeds up model training and provides better network performance in terms of reducing the user request delay and improving the cache hit rate.Meanwhile,the proposed model training framework allows the user data to stay in the local F-AP to complete the model training,which protects the data privacy of users.Finally,the communications efficiency problem of cooperative caching method based on FRL is investigated.FL protects user data privacy at the cost of huge communications resource consumption.To address the problem,three key improvements are proposed to improve the communications efficiency of FRL.First,periodic model aggregation is performed for cooperative trained DRL models.The communications interval of global aggregation is increased to reduce the communications rounds.Second,some F-APs are randomly selected to participate in the global model aggregation.Third,the local models participating in the global aggregation are quantized and compressed.Based on the three key improvements,an FRL with quantizationbased cooperative caching method is proposed.Simulation results demonstrate the superiority of the proposed method compared to traditional caching policies in improving network performance.Also,compared with the traditional FRL algorithm,the proposed method can reduce the transmitted model parameters greatly so that the network load is more lightweight.The proposed method can achieve the same network performance as the traditional FRL algorithm with minimal loss of model accuracy.