Massive Scheduling Method Under Online Situation For Satellites Based On Reinforcement Learning

As the number of satellites grows larger and larger,image satellites play a more and more important role in applications of space-based information,such as military reconnaissance,prevention of disasters,anti-terrorist operations,etc.In these applications,the timeliness of observation requests are very urgent;users usually expect an immediate scheduling.Moreover,observation requests in these applications arrive dynamically,and no information could be obtained beforehand.Due to the requirement of real-time scheduling and the uncertainty of information,the image satellite scheduling in these application is very challenging.The Satellite online scheduling is a real-time scheduling method for situations with high timeliness and dynamic observation requests.Researches about the satellite online scheduling could improve the ability of rapid response of satellites.Therefore,the satellite online scheduling is important in both theoretical research and practical application.Existing researches about satellite scheduling mainly focus on the scheduling problem with certain information.These researches assume information of the observation requests could be fully obtained before scheduling.And the satellite scheduling problem are usually modeled as an optimization problem,then solved with search algorithms.Obviously,such methods could not deal with the situation when the observation requests arrive dynamically.Moreover,in these models,the satellite scheduling is a NP-Hard problem,the difficulty and time consumption grows rapidly as the number of satellites and the number of observation requests grows.To the best of our knowledge,there are no satisfying solution for the satellite online scheduling.Therefore,the problem of satellite online scheduling has been thoroughly studied in this thesis.The main contributions and novelties are listed as follows:1.The key elements of satellite online scheduling are analyzed,and the frameworks about satellite online scheduling on both centralized and decentralized mode are proposed.The framework is the key to solve the satellite scheduling problem.Focusing on the problem of satellite online scheduling,the influences of the key elements in satellites scheduling are thoroughly analyzed,including satellites,observation requests,the networks,etc.Then,two different kinds of organization of satellites are analyzed.And the frameworks about satellite online scheduling on both centralized and decentralized mode are proposed.Unlike the traditional frameworks,this framework consider the satellite as provider of observation services.And the observation requests are regarded as dynamical service requests.The mission of online scheduling is assigning the proper satellites to the dynamical service requests in a real-time style.Apparently,this frame is more suitable for satellite online scheduling compared with existing frameworks of scheduling.2.A satellite online scheduling model based on the Markov decision process is built,and a real-time satellite online scheduling method is proposed.A satellite online scheduling model based on the Markov decision process is built in this thesis: the arriving of observation requests are regarded as a series of random events;the online scheduling are considered as a Markov decision process,and the decisions in this Markov decision process are made immediately when the observation requests arrive.Thus,the online scheduling problem are converted to how to get the optimal scheduling policy.Due to the complication of scheduling,it is very difficult to get optimal scheduling policy directly.So,the scheduling policy is parameterized with a neural network.And an improved reinforcement learning algorithm is introduced to learn the optimal scheduling policy from the scheduling history.The results of numerical simulation shows: the scheduling method proposed in this thesis is a realtime scheduling method with respect to the observation profit.Besides,this method could adapt the change of scheduling situation.3.A distributed satellite online scheduling model based on the Markov game is built,and a real-time distributed satellite online scheduling method with very low requirements of communication is proposed.Due to the low bandwidth of space-based network,the policy-sharing method is adopted to solve the distributed satellite online scheduling problem.And a distributed satellite online scheduling model based on the Markov game is built.Based on this model,the distributed satellite online scheduling is modeled as Markov game with multi-player and multi-stages.Then,this thesis proposes a real-time distributed satellite online scheduling method based on multi-agent reinforcement learning to get the optimal scheduling policies.This method could get the optimal scheduling policies from the scheduling histories of satellites so that it allows the satellite to schedule without massive communications.The results of numerical simulation shows this method could adapt to different scale of distributed satellite scheduling problem,and this method is real-time method with respect to the profit.