Multi-Agent Deep Reinforcement Learning For Flexible Job-Shop Scheduling Problems

With the social development,product demands tend towards diversification,and the production model of the manufacturing industry tends to form a long tail production model that focuses on personalized small batch production and includes various production batch levels as well.With the improvement of hardware performance and the application of AI technology,manufacturing system tends to be more intelligent and gradually forms a deep-fused CyberPhysical Production System(CPPS),which requires both decentralized collaboration among intelligent machines and deep integration of different producing processes.Flexible Job-Shop Scheduling Problem(FJSSP)has always been a core issue towards these manufacturing development trends.The long tail production model requires flexible job-shop scheduling to possess with high flexibility and variable batch splitting capabilities.CPPS requires autonomous scheduling among the decentralized intelligent machines,meanwhile it requires the processes during job-shop producing can be scheduled integratedly.Based on deep reinforcement learning(DRL),this paper studies the FJSSP from four aspects: Variable batch splitting scheduling;Personalized flexible scheduling;Decentralized multi-agents autonomous scheduling;and Scheduling process integration.(1)In order to achieve variable batch splitting scheduling,firstly,a processing-processequipment network containing all feasible processing routes is formed based on the product processing network and the machine capabilities in the flexible job-shop environment.Then,a probabilistic Directed Acyclic Graph(P-DAG)model is constructed based on graph theory,transforming the FJSSP into a graph topology prediction problem,allocating the types and quantities of processing tasks on each machine according to the connection probability of edges in P-DAG,thereby achieving variable batch splitting adaptive scheduling;Secondly,the PDAG model is mapped as a Markov Decision Process(MDP),and a flexible job-shop scheduling model based on reinforcement learning is constructed.Based on this model,DRL algorithms are studied for better scheduling performance under the personalized and highly flexible job-shop environment.Actor-Critic algorithm is employed for better adaptation for dynamic stochastic environtment;Graph Convolutional Networks(GCN)is adopted for critic feature extraction,which enhances both computation efficiency and generalization ability;under a Generative Adversarial Networks(GAN)construction,combining Long Short-Term Memory(LSTM)with RL can fully enroll the history data during scheduling,thus enhancing accuracy and stability.By comparing simulation experiments and ablation analysis,it is demonstrated to possessed with dynamic response ability,generalization ability,and variable batch splitting scheduling ability.(2)In response to the multi-agent decentralized autonoumous trend of FJSSP,a multiagent collaborative flexible job-shop scheduling algorithm based on the P-DAG reinforcement learning model is proposed,achieving autonoumous scheduling between multiple machine agents and multiple job agents.The job agent adopts a Deep Deterministic Policy Gradient reinforcement learning algorithm(DDPG)under the centralized training distributed execution(CTDE)architecture,which performs multi-agent collaborative scheduling in a decentralized partially observable(Dec-PO)environment;The machine agent sorts the jobs waiting to be processed based on an improved job queue rule;To fully explore the behavioral interaction relationships among agents,a GCN-based global behavioral feature extraction algorithm is proposed,which helps to express the multi-agent interaction relationships with lightweight data,improving the collaborative ability and improving algorithm efficiency.(3)Using the results of multi-agent autonoumous scheduling in flexible job shops as the problem prototype,the material transportation by multiple Automatic Guided Vehicle(AGV)is studied.Based on MARL,a self-organized transportation algorithm including multiple AGV agents and multiple transportation task agents is proposed.Based on game theory,a binary competition model for transportation task agents and a Cournot model for AGV agents are constructed,fully considering the impact of competitive interaction between the homogeneous agents;Adopting an improved bilateral matching algorithm enables better collaboration between heterogeneous agents;In order to fully explore the interaction relationships among agents,a Self-Attention Network(SA)with dual views is proposed,improving collaboration ability and computational efficiency.Furthermore,according to the matching results between transportation tasks and AGVs,the multi-AGV collaborative path planning algorithm in flexible job-shop environment is studied.Firstly,a reinforcement learning path planning algorithm based on Rapidly-exploring Random Trees(RRT *)is proposed.In order to improve obstacle avoidance ability and algorithm stability in the Dec-PO environment,a GANstructured SA and RR combined path planning algorithm(GSARL)is proposed,which combines SA network and RR algorithm under the GAN architecture.(4)A multi-AGV flexible job-shop integrated scheduling algorithm is proposed to meet the process integration trend of CPPS.The edge features that representing the transportation status of multiple AGVs in the flexible job-shop are add to the P-DAG model,forming a P-DAG integrated scheduling model for multi-AGV FJSSP.Therefore,the transportation capacity of AGVs is considered at the beginning of FJSSP;Based on the P-DAG integrated scheduling model,the proposed multi-agent collaborative scheduling algorithm is integrated with the multi-AGV collaborative transportation algorithm to achieve multi-AGV flexible job-shop integrated autonoumous scheduling.