Research And Application Of Distributed Computing Resource Scheduling Based On Mesos

Modern astronomical telescopes produce very large amounts of data and grow rapidly.So how to deal with these data rapidly and efficiently has become a major problem in astronomical research.In order to solve this problem,China's astronomical scientists put forward the Mingantu ultra-wide spectrum radio camera(MingantuU Spectral Radioheliograph,MUSER).Based on the requirements of MUSER scientific data processing system,this paper combines Mesos an system of emerging distributed cluster resource management to solve the problem of astronomical data processing in MUSER.Mainly completed the following three points:First,from the needs of distributed computing in MUSER,and based on the requirement of distributed computing in MUSER,this paper analyzes the requirement of MUSER in data processing in distributed cluster,and then designed a distributed framework,namely OpenCluster,for massive astronomical data processing in the calculation.Secondly,based on the design of distributed framework OpenCluster,this paper analyzes the problem of scheduling in distributed cluster of MUSER,then the application of Mesos in the distributed cluster of MUSER is realized based on the key technology of Mesos,so as to completing the development of mesos cluster scheduling mode in MUSER.Thirdly,an improved M-H-DRF algorithm based on DRF algorithm is proposed to solve the defects of DRF algorithm in MUSER distributed cluster.The performance of this algorithm is compared and tested,which proves that this optimized algorithm has good effect in MUSER distributed cluster.Overall,this paper combined with the needs of MUSER massive astronomical data processing,and developed a lightweight distributed framework OpenCluster.Then combined with mesos to design mesos cluster scheduling mode.Finally,the scheduling algorithm DRF of Mesos is improved to make it more suitable for MUSER distributed cluster.In a word the research and application of Mesos in the radio astronomy data processing system will provide a good protection for our astronomical science research.