Research On Multi-scene Edge Cache Strategy Based On Deep Reinforcement Learning

As a new computing paradigm,edge computing can sink part of the computing function and storage capacity of the cloud to the edge network,so that some user requests can be processed on the edge server,improving the computing efficiency and service quality.At the same time,with the explosive growth of mobile data traffic,edge cache,as an important application of edge computing,can cache popular content to edge servers,enabling users to obtain requested resources more quickly and improving the quality of user experience.Due to the limited cache resources of the edge server,how to design a cache strategy that conforms to the edge cache scenario is the focus of research.A good edge cache strategy can effectively improve the cache hit ratio and reduce the delay.Therefore,this paper focuses on the edge cache strategy,and its main work is as follows:(1)The edge cache strategy in the scenario of single edge node is proposed.Existing edge cache strategies based on deep reinforcement learning model the cache decision problem as a Markov decision process,but most algorithms do not constrain the action space,which increases the difficulty of the cache decision network to select the optimal action.To solve this problem,the edge cache algorithm DQNAM(Deep Q-network with Action Mask)is proposed in this paper.An action mask layer is added to the Q Network,which can filter invalid actions according to the current environment state and request information,narrow the action space,and improve the applicability and performance of the cache model.In addition,in view of the fact that the content requested by the current users is mostly streaming media resources such as video and audio,this paper takes the content retention rate as the content popularity,and makes content popularity prediction.Then,according to the content popularity,the action space and reward function in the Markov decision-making process are designed.(2)The edge cache strategy in the multi-edge node scenario is proposed.In the multinode scenario,the edge cache node can only obtain local status information.If the cloud server is used to collect global information and make caching decisions,the communication overhead and caching efficiency will be increased.To solve this problem,this paper proposes the edge cache algorithm MADGS(Multi-Agent Deep deterministic policy gradient with Gumble Softmax),which is a multi-agent reinforcement learning algorithm with centralized training and decentralized execution.The algorithm is improved on the multi-agent depth deterministic strategy gradient algorithm.By reparameterizing after the cache decision network,the output discrete action is derivable and can participate in gradient updating.After MADGS training is completed,each node can make cache decision according to its own state through the local cache decision network,which reduces the communication overhead and improves the cache efficiency.(3)This paper uses the video data of Kuai Rec data set as the cache content,and compares the proposed edge cache strategy with other cache strategies under different conditions.Experimental results show that the cache strategy proposed in this paper can effectively improve the cache hit ratio and reduce the delay.