Containerized Network Function Deployment Optimization Via Meta Reinforcement Learning

With the development of virtualization technology,containers have been proposed as a compelling alternative or supplement to Network Function Virtualization(NFV)to provide light-weight and easy-deploying network functions.For cloud service providers,the different placement strategies of Containerized Network Function(CNF)lead to different Quality of Service(Qo S)and OPerating EXpense(OPEX),which will affect the overall performance of CNF platforms.Considering the dynamic and diversity of user demands in the practical network environment,how to dynamically deploy CNF and provide on-demand network services is becoming an urgent and significant topic for CNF performance optimization.To minimize the long-term OPEX,the CNF deployment problem is first formulated into a mixed-integer linear programming(MILP)form with the consideration of the dynamical user requests in complex network environment,and then proved to be NP-hard.To deal with the computational complexity,this MILP problem is further modeled into a Markov Decision Process(MDP).Inspired by the existing researches of meta reinforcement learning in other fields,a dynamic deployment algorithm for CNF based on MetaAdvantage Actor-Critic(Meta-Critic)is proposed.To further improve the overall performance of CNF platform,a Model-Assisted Meta-Critic(m Meta-Critic)algorithm is proposed,which uses the mathematical model to assist the pre-training of Meta-Critic to realize rapid CNF deployment decision-making in dynamic network environment.Experiments in different network scales and task scenarios verify the effectiveness and stability of the m Meta-Critic algorithm for CNF deployment.Compared with the MetaCritic and A2 C,the m Meta-Critic can achieve faster convergence,higher reward,and lower costs.To be specific,the maximum convergence speed is increased to 89.80%,and the reward under 5,000 rounds is increased to 83.42%.Besides,compared with the existing heuristic algorithms such as Greedy and Instant,the m Meta-Critic can achieve lower cumulative OPEX.