Research On Industrial Edge Cache Replacement Strategy Based On Deep Reinforcement Learning

As the Internet of Things(Io T)technology is widely applied in industrial applications,an increasing number of field smart devices are deplied in industrial field,forming a large-scale industrial Internet system.Edge computing,an emerging computing paradigm,is expected to become an important technological means to realize intelligent manufacturing.It distributes computing and storage resources to network edgies,alleviating high transmission latency caused by limited bandwidth in cloud computing.However,the computing and storage capabilities of edge nodes are often limited.Therefore,it is necessary to adopt a reasonable caching strategy to optimize the data transmission efficiency.This thesis focuses on industrial edge caching strategies based on deep reinforcement learning algorithms.The main research contents are as follows.(1)The edge cache problem model is fomulized for typical industrial applications.To meet the diverse real-time requirements of industrial control applications,the overall cache hit rate and the control file cache hit rate as dual optimization objectives are considered.The cache replacement process is then modeled as a Markov decision process,and the reward function is defined considering the different real-time requirements of control files and best-effort files.(2)A cache replacement strategy based on the Deep Recurrent Q-Network(DRQN)algorithm is proposed to adapt to the dynamic popularity of file requests.The Long Short-Term Memory(LSTM)neural network is introduced into Deep Q-Network(DQN)to estimate Q values more accurately,and can adapt to dynamic popularity.Simulation experiments compare the performance of the DRQN algorithm and the reference algorithms under different cache capacities,and under static and dynamic popularity request sequences.The experimental results show that the proposed cache replacement strategy based on DRQN can effectively improve the cache hit rate of industrial control files without sacrificing the overall cache hit rate.(3)A cache replacement strategy based on Recurrent Deterministic Policy Gradient(RDPG)is proposed for large-scale file number.The algorithm uses the Wolpertinger architecture to map the continuous action space to a discrete action space and reduces the high computational complexity caused by the large discrete action space by using a candidate action set.An optimization method of cache file ordering is proposed,to solve the weak correlation between discrete actions.Simulation experiments are conducted to compare the performance of the RDPG algorithm and reference algorithms for both static and dynamic popularity request sequences.The results show that the improved RDPG-based edge cache replacement strategy can improve the overall cache hit rate and effectively reduce computational complexity in large discrete action spaces.