Application Of Markov Decision Process In Wireless Caching Networks

With the development of wireless communication technology,wireless transmission rate is getting faster.People's demand for wireless data transmission is also increasing.At the same time,content-centric data(video,audio,etc.)has gradually become the mainstream of wireless data transmission.Wireless cache technology is to store these content-centric data in cache nodes at the edge of the network,thereby improving the overall performance of the network.In this paper,the scheduling of downlink file transmission in one cell with the assistance of cache nodes with finite cache space is studied.Specifically,requesting users arrive randomly and the base station(BS)reactively multicasts files to the requesting users and selected cache nodes.The latter can offload the traffic in their coverage areas from the BS.We consider the joint optimization of the abovementioned file placement and delivery within a finite lifetime subject to the cache space constraint.Within the lifetime,the allocation of multicast power and symbol number for each file transmission at the BS is formulated as a dynamic programming problem with a random stage number.Note that there are no existing solutions to this problem.We develop an asymptotically optimal solution framework by transforming the original problem to an equivalent finite-horizon Markov decision process(MDP)with a fixed stage number.A novel approximation approach is then proposed to address the curse of dimensionality,where the analytical expressions of approximate value functions are provided.We also derive analytical bounds on the exact value function and approximation error.Based on the expression of approximate value function,this paper presents a low complexity online resource allocation algorithm.The approximate value functions depend on some system statistics,e.g.,requesting users' distribution.One reinforcement learning algorithm is proposed for the scenario where these statistics are unknown.Numerical simulations show that the low-complexity algorithm based on the approximation function proposed in this paper can significantly reduce the average transmission cost of the base station compared with some benchmark schemes.