Research On Optimal Control Of Nonlinear System Based On Adaptive Dynamic Programming And Its Application In Microgrid

Optimal control can not only ensure the closed-loop stability of the system,but also can optimize some performance indexes of the system.The classic dynamic programming method used to solve the optimal control problem has certain limitations,such as the “dimensional disaster” problem.In order to overcome this problem,a new method to approximate the optimal controller is emerging in the field of optimal control,namely adaptive dynamic programming.This method combines the ideas of adaptive critic design,neural network,and reinforcement learning to obtain an approximately optimal closed-loop feedback control law,so it is regarded as an effective method for solving optimal control problems.However,most of the complex industrial systems such as micro-grid systems with multiple inverters connected in parallel exhibit nonlinear characteristics and often have a time lag phenomenon.Meanwhile,they usually require the cooperation of various subsystems to solve the problem,that is,it has the characteristics of group cooperation.Therefore,the research on optimal control of nonlinear time-delay systems and nonlinear multi-agent systems is urgently needed to provide important theoretical support for practical systems.Based on the adaptive dynamic programming method,this dissertation deeply studies and solves the optimal control problem of nonlinear time-delay systems and the optimal coordinated control problem of nonlinear multi-agent systems.The adaptive dynamic programming method is applied to solve the distributed optimal coordinated control problem of the microgrid.The main research points of the dissertation can be briefly described as:(1)For the problem of optimal tracking control for a class of continuous nonlinear time-delay systems,a value iteration algorithm based on integral reinforcement learning is proposed,and Lyapunov theory is used to prove the convergence of the algorithm as well as the stability of the system.For the implementation of this iterative algorithm,a single-layer critic neural network is used to approximate the optimal cost function so as to obtain an approximately optimal controller,so that the system asymptotically tracks to a given desired trajectory.(2)A data-based distributed adaptive dynamic programming method with identifier-critic architecture is proposed for the optimal coordination control problem of continuous-time nonlinear multi-agent systems with completely unknown dynamics.First,an online adaptive identifier is developed to estimate the unknown system dynamics,and simultaneously a critic neural network is employed for approximation of the optimal cost function,which yields approximated optimal coordination control in real time.Finally,the Lyapunov theory is used to prove the stability of the system.(3)A novel adaptive dynamic programming approach is proposed to address the optimal leaderfollower consensus issue for constrained-input continuous-time multi-agent systems with completely unknown dynamics.Firstly,the fundamental smooth assumption is not satisfied for the value function by use of vanishing viscosity solutions.Secondly,the control cost functions take a more general form which guarantees the continuity of the optimal control policy instead of the specific integrand form used in previous input constrained reinforcement learning-based schemes.Based on these results,a novel identifier-critic-actor structure is introduced to extend the conventional critic-actor reinforcement learning framework into a distributed model-free one,where the learning of the identifier,critic and actor is online and simultaneous.(4)Considering the parametric uncertainties for microgrids,an adaptive secondary control method via distributed adaptive dynamic programming is proposed for microgrid frequency and voltage stability problem.For the controller design,a single critic neural network is introduced to deal with the uncertainties in the model of distributed generators.By using such technique,the distributed secondary controller can be trained directly from data measurements,and thus can adapt to system or external disturbances.Accordingly,stability of the closed-loop system is proved by Lyapunov technique.Furthermore,since the frequency and voltage information needs to be sampled and transmitted at a high frequency in the design of the distributed controller,in the case of limited communication resources,the manner of such periodic sampling transmission is too conservative,which increases network communication pressure.Therefore,based on the above work,a distributed secondary control method based on event-triggered adaptive dynamic programming is proposed,which is characterized by each distributed generator using its own state information and other information that can interact with it.For the proposed method,the critic network is only updated at the trigger instants decided by the designed event-triggered condition and thus the control signal is transmitted in an aperiodic manner to reduce the computational and the transmission cost on the basis of achieving the control target.