Research On Autonomous Maneuvering Decision Method For Aircraft Based On Deep Reinforcement Learning

The world has witnessed the continuous development of modern warfare in informatization and intelligence.As a critical technology to enhance the intellectual level of air combat,the autonomous maneuver decision-making of aircraft is inevitably drawing more attention from countries worldwide.The modern air combat environment is complex and rapidly changing,making the military research focus on autonomous maneuver decision-making methods that can accurately perceive the air combat environment and yield reasonable decisions in the last decade.Deep reinforcement learning has made breakthroughs in sequential decision problems in recent years and can provide new ideas for the solution of aircraft maneuver decision-making problems.In this thesis,the guidance intelligent maneuver decision-making method of aircraft in three-dimensional space is analyzed from the perspective of deep reinforcement learning on the background of the guidance maneuver of aircraft in three-dimensional.The study aims at providing necessary references and support for the intelligent advancement of aircraft maneuver decision-making method and is carried out in terms of the following aspects:(1)A control method of the basic maneuvering action of the vehicle is designed to solve the problem of an imperfect aircraft guidance training environment in three-dimensional continuous space,which can perform complex tactical actions through continuous multi-step maneuver control and meet the basic needs of the aircraft guidance training tasks.Firstly,an aircraft motion model with both tangential and normal overload as the control quantities is established based on the vehicle dynamics and kinematic model,which provides accurate and effective aircraft state information for the deep neural network.Secondly,the aircraft maneuvering control mode is analyzed and designed based on the pilot’s maneuver process,while the available set of vehicle actions is determined,and the aircraft guidance training environment is improved.(2)An aircraft guidance reward reshaping function is proposed to solve the problems that the intelligent maneuver trajectory of vehicle guidance is not smooth enough and the deep reinforcement learning algorithm is slow in its training speed.The function can guide the aircraft to reach the moving target position from any location in the limited airspace,which means the intelligent maneuver decision-making of aircraft guidance task is realized.Firstly,an adequate representation of the environment state is established by sensing and evaluating the training environment information in the vehicle guidance training environment.Secondly,a reinforcement learning reward reshaping function is designed to improve the training efficiency of the intelligent body and the quality of the vehicle flight trajectory based on the relative position as well as the angle between the vehicle and the moving target.The experimental simulation results show that the aircraft maneuver trajectory and the training speed of the agent can be improved due to the reward reshaping function in the guided maneuver of the moving target.(3)An intelligent air combat simulation system is designed and implemented to solve the problem that the current simulation system can hardly meet the demand for intelligent air combat maneuvering decisions.The system supports deep reinforcement learning intelligent body access.The training and verification of the agent can be carried out in the system,supporting the simulation technology for the intelligent advancement of aircraft maneuver decision-making methods.Firstly,the general architecture of the intelligent air warfare simulation system is designed in terms of the interaction mode between deep reinforcement learning and the environment.Secondly,the intelligent interface of the simulation system is designed,and the way of interaction between the intelligent body and the simulation environment is defined.Finally,the training results of one-to-one air combat intelligence in3 D space using the system are demonstrated.The platform can both display aircraft trajectory and 3D flight posture and train maneuvering decision-making intelligence.