Research On Reinforcement Learning Control Method Of Micro Air Vehicle

Miniature Munition is a type of classical miniature flight vehicle.They are small in size,easy to carry with high precision and little collateral damage.They will play an important role in future wars because widely used by UAV and individual soldiers.The attitude control is an important aspect in the study of miniature munition.Optimal control,a common way of attitude control design,can get the solution of regulator or tracker by minimizing the performance index.However,the accurate system model is usually necessary to be established.While in practical,due to influence of environment and changes in munitions it is hard to obtain the system model or the model may have strong nonlinearity.So it is difficult to get the optimal control policy.In order to explore the application of artificial intelligence in traditional aircraft,the reinforcement learning(RL)is used to get the optimal control policy of miniature munition attitude without requiring the system dynamic.Firstly,the dynamic model of miniature munitions is established.The relationship of absolute derivative and relative derivative is used to get the motion equations.Considering two state variables like attack angle and pitch rate,linear equations of the short-period motion equation is obtained by linearization.The commonly used reinforcement learning methods are discussed and Actor-Critic is used to design controller in the following.Secondly,an adaptive optimal algorithm for linear quadratic problem is developed.A discounted performance function is introduced for the linear quadratic tracking(LQT)problem.A discounted algebraic Riccati equation(ARE)is then derived which gives the solution to the LQT problem.The Actor-Critic is used to learn the solution of discounted ARE online without requiring complete knowledge of the system dynamics.Comparing with the standard method,the simulations show that the optimal tracker and optimal regulator can be obtained by RL method for miniature munition.Meanwhile,the influence of weight in optimal regulator cost function is discussed.Finally,the proposed idea is extended to solve optimal tracking control for nonlinear systems.The input constraints are also taken into account for nonlinear systems.Establishing a non-quadratic performance function,the tracking Hamilton-Jacobi-Bellman equation can get and an off-line control policy is given.Taking advantage of nonlinear approximation ability of the neural network,it is used to solve the value function and obtain an approximate optimal control strategy.The simulation shows the approximate optimal control strategy of miniature munition with saturating actuators.