Logistics-involved QoS-aware Service Composition In Cloud Manufacturing With Deep Reinforcement Learning

Cloud manufacturing is a novel manufacturing model that aims to provide on-demand manufacturing services to consumers over the Internet.Service composition is an essential important issue as well as an important technique in cloud manufacturing（CMfg）.It supports construction of a larger-granularity,value-added service by combining a number of smaller-granularity services to satisfy consumers’complex requirements.Meta-heuristics algorithms such as genetic algorithm,particle swarm optimization,and ant colony algorithm are frequently employed for addressing service composition issues in cloud manufacturing.These algorithms,however,require complex design flows and painstaking parameter tuning,and lack adaptability to dynamic environment.Deep Reinforcement Learning（DRL）provides a new approach to solving cloud manufacturing service composition（CMfg-SC）problem.As a model-free artificial intelligence method,DRL enables the system to learn the optimal service composition solution through training.In this article,we aim to find possible applications of DRL in CMfg-SC.We propose a cloud manufacturing service composition model that considers logistics service quality in Qo S,and use deep reinforcement learning algorithm PD-DQN to find the optimal service composition solution.A cloud manufacturing service composition model is constructed,including single composition task,manufacturing resource services and logistics service,the calculation of Qo S.Secondly,based on the cloud manufacturing service composition model,a Markov decision process is established,i.e.<state space,action space,reward function,state transfer function>.Finally,the PD-DQN is used to solve CMfg-SC.To improve the efficiency of the algorithm,the PD-DQN algorithm is based on a deep Q network model with prioritize replay,and uses a dueling architecture to build a neural network.In order to verify the effectiveness of the PD-DQN algorithm in solving the CMfg-SC problem,a series of experiments were performed.The experimental results show that the PD-DQN algorithm can effectively solve the large-scale cloud manufacturing service composition problem.At the same time,PD-DQN is robust and adaptive for the CMfg-SC problem in a dynamic environment.The PD-DQN algorithm can effectively learn the optimal/near optimal service combination solution for CMfg-SC problems of different scales,and has better scalability.Finally,we use the Wilcoxon rank test to analyze PD-DQN,DQN,and Q-Learning respectively.When the scale is 30³⁰,15 groups experiments are performed to obtain statistics of 114 and 120,which proves that PD-DQN can effectively solve CMfg-SC problem.