The Hierarchical Decision-making And Control System Of Aerial Robots

In the past decade,drones based on multirotor structures have played an increasingly impor-tant role,such as power line inspection,police operations,photography and filming,entertainment and other scenes.With its simple structure,flexibility and easy-to-control,the multi-rotor drones have been introduced in many application scenarios and have become one of the important tools in real-life.At the same time,however,most multi-rotor drones still work manually.At the middle level(planning)and the upper level(decision-making)which basically rely on human operations,there is still lack of corresponding intelligent algorithms.It is difficult for human to make effective decisions in complex and dynamic environments.Reinforcement learning is a decision-making algorithm that has developed rapidly in recent years.For the sequential decision making problems in model-free environments,which are difficult to handle by traditional optimization algorithms,reinforcement learning can provide more intuitive modelling and solving methods.At present,it has achieved a great success in the field of computer games.Applying it to the decision-making level of a multi-rotor aerial robots requires a certain degree of consideration in the decision level to the middle and lower levels of the controller.For example,the probability that the middle and lower controllers successfully complete the decision,and the time required to complete the action.This is a challenging topic that combines the area of artificial intelligence with the area of robotics and control theories.In this paper,the problem of driving multiple ground moving targets by a single aerial robot is taken is the targeting scenario.The seventh generation mission of the International Aerial Robotics Competition(IARC)is the specific application,in which the hierarchical decision and control algorithms are applied.The main contents and research results of this paper include the following five aspects:1.For an aerial robot interception the ground target ground mobile robot,it needs to dynamically replan the trajectory according to the continuously updated target position.In order to successful interception,this paper proposes a three-stage planning method,which divides the whole process into three separate stages:cruise,track and approach.For each stage,a sliding window quadratic dynamic programming method is proposed to generate the control command to realize the interception of the ground moving target.2.Filtering and prediction of the target location is also important.The traditional filtering method relies on the accuracy of modeling.The modeling error causes error in the filtering result,and the traditional filtering framework can hardly solve this problem well.We propose a method of integrating neural networks into the models to improve prediction results.In this method,it is an iterative process to generate filtering results,to train the network with filtering results,and to update the system model with the trained network.After integrating the neural network,the filtering and prediction results have been significantly improved.3.Concerning the complexity,risks and difficulties of aerial robot experiments,the simulation environment is important and necessary tool.For a simulation system we propose four requirements:dynamics and collision engine,multi-agent control,sensor simulation,visualization of simulation results.We combine the ROS and Gazebo to achieve the above four requirements for aerial robot simulation.Furthermore,in order to verify the time prediction model in shepherd game,we use the path integral control in the simulation system and verify the time prediction model.4.For the top-level decision-making problem of the shepherd game,we model it using the Markov decision process.Since this problem is a discrete decision problem in a continuous time and continuous state multi-agent system,there is a problem that the decision result(action)needs a varying duration of time to complete.We call it the action delay.By correlating the action delay with the reward by the known model,and discretizing the continuous state into a lookup table,we further generate the corresponding action reward by randomly sampling the initial state from the lookup table.5.Furthermore,through the systematic analysis of the shepherd game,we integrate the problem of reinforcement learning with action time.This problem can be solved by establishing a hierarchical system,combining the upper decision output with the underlying control set value input,training the time prediction model in advance and integrating it into the reinforcement learning framework.Based on this,we propose a framework for hierarchical reinforcement learning combined with the underlying controller,which creatively solves the problem of discrete event control in a continuous-time system.The research scenario of this paper mainly focuses on the seventh generation task of the IARC-the shepherd game,and several parts of this paper are applied to the robots of author’s team.The results of these researches in IARC mainly include:1).In 2016,the technical report of decision-making research was submitted to the organizing committee,prompting the organizing committee to modify the competition rules and reduce the requirements for the final completion of the task;2).In 2016-2018,the team applied the three-stage trajectory planning method to intercept the ground target and achieved a very high success rate;3).In 2017,based on the research results of this paper,the time-based decision maker was programmed for the first time in the participating aerial robots,and successfully achieved 3 scores in that year,ranked first in the participating teams around the world;4).In 2018,our team implemented the proposed framework and finally ended the seventh generation mission,becoming the 7th world champion of the IARC.