Research On Hierarchical Deep Reinforcement Learning Algorithm Toward Carrier-based Aircraft Automatic Landing Problem

In recent years,with the development of military technology,carrier-based aircraft has played an increasingly critical role in it.The landing process is a very important step in the aircraft carrier/carrier-based aircraft system.Due to many uncertain factors in the early manual assisted landing system,the automatic landing technology gradually emerged.Reinforcement learning has been applied in many fields in recent years,and its excellent learning and decision-making capabilities enable it to perform well in a variety of task types.This thesis will study the application of reinforcement learning in the automatic landing process of carrier-based aircraft in the virtual environment X-Plane.We hope that this thesis can do some forward-looking work for artificial intelligence algorithms that can be more generally applied in the military field.In the prior work,we have completed the study of the landing process of the carrier-based aircraft in the basic landing task.The characteristic of the basic landing task is that the state of the carrier-based aircraft before landing is within error range of the standard state.In the basic landing task,the carrier-based aircraft agent can easily explore rewards and train the model.This thesis focuses on the landing task under abnormal conditions.Abnormal state means that the initial state of carrier-based aircraft is not within error range of the standard state,which makes exploration difficult and unable to obtain enough rewards to train the reinforcement learning model.This thesis first proposes a hierarchical reinforcement learning algorithm based on the policy dimension.The algorithm model is composed of two layers of policy,the lower policy is a subtask controller,which is composed of several subtask agents.The subtask agents can independently complete posture adjustment,heading control,speed adjustment,approaching the glideslope and basic landing tasks.The higher policy is responsible for scheduling and coordinating the lower subtask agents to complete the entire landing process.This thesis completes the design of subtasks in the lower policy,the modeling of Markov decision process(MDP)of subtasks,and the training of subtask agents.On this basis,we have completed the design and implementation of the hierarchical model.The trained hierarchical model can control carrier-based aircraft to complete the landing process under abnormal conditions.Although this hierarchical reinforcement learning algorithm based on policydimension can guide carrier-based aircraft to complete the landing process.However,when the initial distance between carrier-based aircraft and landing target point is large,the performance is not very stable due to the delay of the reward.Therefore,this thesis further proposes a hierarchical reinforcement learning algorithm based on task dimension,which divides the landing task into main control task and auxiliary task.When carrier-based aircraft is in an abnormal state,carrier-based aircraft needs to first correct itself to the standard state,and then landing alongside the glideslope.The auxiliary task refers to the process of carrier-based aircraft correcting from the abnormal state to the standard state.The main control task refers to the process of carrier-based aircraft landing along the glideslope smoothly.This thesis designs and implements the auxiliary task and the main control task respectively.Then we connect the auxiliary task with the main control task to ensure that carrier-based aircraft can finish landing process successfully in abnormal conditions.In order to verify the correctness and effectiveness of the algorithm proposed in this thesis,this thesis builds a training platform in the X-Plane environment and applies two hierarchical reinforcement learning algorithms to the landing process of carrier-based aircraft under abnormal conditions.The experimental results show that the two algorithms proposed in this thesis can control carrier-based aircraft to complete landing process,and the hierarchical reinforcement learning algorithm based on task dimension is more stable in the entire landing process.