Research On Distributed Cache Technology Based On Reinforcement Learning In Wireless Converged Networks

In recent years,with the popularity of wireless intelligent terminals,the emergence of various multimedia services has led to an exponential increase in mobile data traffic.This brings great pressure and challenge to the communication network.Data offloading technology based on wireless converged networks is an effective solution.However,the efficiency of centralized caching mechanism in traditional wireless converged network needs to be further improved.Therefore,this paper introduces the distributed cache technology in wireless convergence network,and focuses on the distributed cache technology based on reinforcement learning.First,in the case of a distributed two-hop cache network,the problem of how to maximize the equivalent throughput of the wireless converged network through resource scheduling under the constraints of cache storage and bandwidth of a router-node is studied.Since it is great challenge to obtain a tractable solution by the classical optimization algorithm,the reinforcement learning method is adopted to solve it.By describing the service scheduling problem as a Markov Decision Process,a reinforcement learning scheduling algorithm based on deep deterministic policy gradient and a scheduling algorithm combining dynamic programming and a deep Q-network are proposed.Then,the simulation is carried out under the user’s uniform distribution,non-uniform distribution and extreme distribution,and compared with existing cache schemes and centralized cache scheme in terms of equivalent throughput,latency,cache utilization and load balancing coefficient.The simulation results show that the proposed reinforcement learning scheduling algorithm based on deep deterministic policy gradient has the best performance in terms of equivalent throughput,followed by the scheduling algorithm combining dynamic programming and a deep Q-network.But both are better than existing cache schemes and centralized cache scheme.The equivalent throughput increases by 5.5% to 8% in the case of uniform and non-uniform distribution of users.However,it can be seen from the delay and load balancing coefficients that the proposed algorithms come at the expense of latency,load balancing,and complexity.At the same time,the proposed algorithm improves the equivalent throughput even more when the users are unevenly distributed,and it is more used in practical scenarios.Then,this paper expand the distributed two-hop network into a distributed multi-hop network.By redefining the evaluation index of the network,and further establish the optimal equivalent throughput model of the distributed multi-hop network under the wireless converged network.Based on the two scheduling algorithms of the distributed two-hop network,combining the advantages of both,a reinforcement learning scheduling algorithm based on dynamic programming and deep deterministic policy gradient(DPDDPG),and odd-even based dynamic programming and deep deterministic policy gradient(ODP-DDPG)scheduling algorithm are proposed.The simulation results show that in the case of uniform distribution of users,the equivalent throughput of the network increases by 18% and 2.4% respectively with the increase of the number of hops in the distributed network.In the case of non-uniform distribution of users,the equivalent throughput of the network increases by 19% and 3% respectively.And the corresponding delay also increases by 0.5 hops,but the increase is not large.The proposed algorithm can better schedule services in the distributed multi-hop network to improve the equivalent throughput of the network.In addition,compared with DPDDPG,ODP-DDPG has a better load balance with a very small equivalent throughput cost.Finally,we summarize the existing research work and propose a prospect for the next study.