Maintenance Optimization Of Complex Engineering Systems By Deep Reinforcement Learning

With the rapid development of science and technology,various complex systems have penetrated into human social life.The failure of systems may cause a serious social impact or even threaten the safety of human life.As an important method to guarantee the performance capacity and extend the lifetime of systems,maintenance activities and the corresponding maintenance decision problems have attracted significant concern.As modern advanced engineering devices and systems are designed towards larger size,more complex,and higher precision,the challenges to maintenance decision-making are presented.Meanwhile,in many engineering practices,maintenance resources,such as maintenance budgets,maintenance materials,maintenance time,maintenance personnel,maintenance equipment,and maintenance space,are usually limited,and the traditional maintenance decision methods with unlimited maintenance resources are intractable for maintenance decision-making problem with limited maintenance resources.Hence,developing maintenance decision-making with limited maintenance resources is a pressing necessity.In recent years,with the rapid development of artificial intelligence,deep reinforcement learning methods have emerged as a powerful method for solving largescale complex sequential decision problems.Based on the advantages of deep reinforcement learning and heuristic optimization methods,this dissertation is to address the critical challenges in maintenance decision problems for engineering systems.The primary research contributions and innovative outcomes are summarized as follows:(1)An optimal maintenance strategy for multi-state systems with single maintenance capacity and arbitrarily distributed maintenance time is developed.For systems that can continuously operate while some components are being repaired,a maintenance optimization problem with a single maintenance capacity is studied.In contrast to the exponential assumption for the distribution of maintenance time in most reported works,the time for each maintenance action can be arbitrarily distributed.by introducing decision epochs,the embedded Markov chain is constructed to model the state transition process of a system.Under a limited maintenance budget,two optimization problems are formulated by treating either the stationary availability or the expected performance capacity of a system as an objective.A customized genetic algorithm with the mapping of decision variables is utilized to resolve the resulting optimization problems.As demonstrated by an illustrative example,the proposed methodology can obtain the optimal maintenance strategy,and the customized genetic algorithm is more efficient and robust compared with the traditional genetic algorithm.(2)A sequence planning method for selective maintenance of multi-state systems under stochastic durations of breaks and maintenance actions is developed.In many industrial and military environments,the durations of breaks and maintenance actions can be uncertain,and the sequence of maintenance actions can influence the number of completed maintenance actions in the break between adjacent two missions,influencing the success of the next mission.A selective maintenance model is proposed to identify the optimal sequence of selected maintenance actions.The saddlepoint approximation is utilized to facilitate the computation of the involved multi-dimensional integration in evaluating the probability of a system successfully completing the next mission.As the sequence planning process is naturally accordant with the behavior of an ant visiting multiple nodes in order,the ant colony optimization is tailored to search for the optimal solution.As demonstrated in two illustrative examples,the proposed method can identify the optimal maintenance sequence in a computationally efficient manner.(3)A deep reinforcement learning approach to dynamic selective maintenance optimization for multi-state systems with multiple missions is developed.Cases of multiple consecutive missions are oftentimes encountered in engineering practices.Maintenance actions can only be performed during the break to restore the aged system to a better condition and thus complete as many future missions as possible.Under limited maintenance budget and maintenance time,a selective maintenance optimization for multi-state systems that can execute multiple consecutive missions over a finite horizon is developed.A discrete-time finite-horizon Markov decision process with a mixed integer-discrete-continuous state space is formulated.Based on the framework of actorcritic algorithms,a customized deep reinforcement learning method is put forth to overcome the “curse of dimensionality” and mitigate the uncountable state space,and a postprocess is developed for the actor to search the optimal maintenance actions in a large-scale discrete action space.As demonstrated in the two illustrative examples,the proposed DRL approach can dynamically identify the optimal maintenance actions for all the components in a computationally efficient manner.(4)A cumulative performance-oriented optimization method for dynamic loading strategy of repairable multi-state systems is developed.Traditional studies on multi-state systems tend to focus on the system performance capacity at a specific time instant.However,the cumulative performance of a system over the mission time also has important engineering significance.From a cumulative performance perspective,a dynamic performance rate(load)optimization problem is investigated.The state transition intensity model of each component is defined as a function of the number of the conducted imperfect maintenance actions and the performance rate of the component.The degraded components in a system are dynamically maintained to recover to their better conditions.To achieve the maximum expected cumulative performance within a finite time horizon and limited maintenance resources,a Markov decision process with a continuous action space and a mixed integer-discrete-continuous state space is formulated.The deep deterministic policy gradient algorithm is customized to overcome the “curse of dimensionality” and mitigate the uncountable state and action spaces.As shown in the two illustrative examples,the customized DRL method can efficiently identify the optimal dynamic loading strategy with acceptable accuracy and runtime.(5)A dynamic scheduling method of inspection and maintenance for multi-state systems under time-varying demand is developed.Traditional multi-state system maintenance decision methods tend to assume that user requirements are time-invariant,whereas,in many industrial and military environments,systems need to meet timevarying user demands.This dissertation proposes a joint optimization model for dynamic inspection and maintenance of multi-state systems under time-varying demands.Nonperiodic inspections are performed on the belief states of components,and maintenance actions are dynamically scheduled based on the inspection results,so as to minimize total cost.By introducing the concept of decision epoch,the resulting problem is formulated as a Markov decision process with mixed discrete-continuous state space.To cope with the “curse of dimensionality”,an agent is constructed based on the actor-critic framework,where three extra input features and a stochastic policy for the agent are formulated.The agent is trained by the proximal policy optimization method.Two illustrative examples showed that the proposed method is effective and efficient in dynamically scheduling the inspection and maintenance strategy for different types of time-varying demands.