Reinforcement Learning With Its Application In Coplanar Air Combat

As a main tool to deal with pursuit-evasion games, the differential game theory has developed very well for fifty years. However, it is difficult to apply the results in real air games. Most analytical studies need precise mathematics model and involve to solve the problems of nonlinear two-point boundary value and singular surfaces, which are formulated by the set of necessary conditions of game optimality. So it is impossible to get the accurate solutions.With the development of artificial intelligence, many recent studies have been devoted to combining intelligent control with differential game theory. In the realization of intelligent control, it must be involved in automatic acquisition and utilization of knowledge. As one of machine learning, reinforcement learning not only has this function but also can expand the acquisitive resource.By using the method of combining reinforcement learning and differential game theory, the coplanar air combat problem between two aircrafts is analyzed. Based on this method, it is avoided to solve the tedious two-point boundary value problem derived from the optimal control theory. By the human fuzzy logic, the rule of air-combat policy is built, which decomposes the state space , decreases the action space and improves the efficiency of neural network.The value function approximation of reinforcement learning with neural network is studied. Based on this method, the problems of the "curse of dimensionality" in the reinforcement-learning algorithm and "structure credit-assignment" in learning are solved.In the simulation, many practical conditions and realistic aerodynamic data are analyzed. The simulation results show the validity of applying reinforcement-learning-based differential games to coplanar air combat.This paper is outlined as follow: the importance and method of research in the two-aircraft combat is firstly analyzed, And then the general structure and the bases of design are presented. In Section 2, the nature, history and algorithms are introduced. In Section 3, the rule of air combat countermeasure is given, and the intelligence guidance problem of air combat withreinforcement learning is discussed. In Section 4, Based on the horizontal two-aircraft combat dynamics model, the numerical simulations are made respectively for constant-speed and variable-speed cases, and then these results are analyzed.