Research On Cluster Computing Of Distributed Astronomy Application Based On Docker

Construction of the Square Kilometer Array(SKA)is about to begin and the subpackages will enter the critical design assessment phase.The Scientific Data Processor(SDP)is one of the important work packages of SKA.With the continuous progress of construction,facing the upcoming massive astronomical data,improving data processing capacity,reducing energy consumption and rapid deployment of astronomical applications become the major problems to be solved.Among these,rapid automatic deployment of astronomical applications has become the primary problem because of it provide effective support for improving data processing capabilities while helping to reduce energy consumption.Aiming the primary problem and considering that the container cloud based on cloud and container technology is the platform technology that SDP may adopt in the future,this paper studies the distributed astronomical application cluster computing based on Docker on the container cloud platform.The research is centered on the MPIbased distributed technology for astronomical mass data processing,and the commonly used visibility function calibration software SAGECAL is the concrete research object.The realization of automatic deployment and the optimization of load balancing are carried out respectively.The specific research work of this paper includes:(1)The automatic deployment method of distributed astronomical application cluster based on MPI is studied.Using scripts and Docker Swarm,an automatic deployment framework for astronomical application clusters is designed and implemented,which can effectively support distributed computing using MPI.By providing an interactive and user-friendly interface,users can complete cluster computing,terminate and delete clusters,and enter the cluster node to do other operation with less necessary parameters.The deployment module will choose scheduling algorithm based on multi-objective optimization or batch scheduling scheduling algorithm according to the cluster node size to adapt to the single application and cluster mixed deployment scenario in the process of automatic deployment.Experimental results show that the automatic deployment method proposed in this paper greatly improves the deployment efficiency of distributed astronomical application clusters.(2)The batch scheduling algorithm based on genetic algorithm is studied,which takes the load balancing of astronomical application cluster as the optimization goal.The main research content is that all nodes of the astronomical application cluster will be deployed as a whole in the dynamic load environment of the physical machine cluster,through evolutionary algorithms to find the optimal solution of the deployment scheme.Evolutionary algorithm is a "cluster of algorithms",which contains a variety of specific algorithms can explore the optimal solution of the function.In order to provide a reliable guarantee for the future use of SKA-SDP,this paper selects the evolutionary genetic algorithm as the basis to carry out the research of batch scheduling algorithm.Based on the reliability requirements of SKA-SDP for massive astronomical data processing,the batch scheduling algorithm studied in this paper is closely around the absolute convergence of the guarantee algorithm and avoiding falling into the local optimal solution,which are the two core requirements.The implementation of batch scheduling algorithm uses elitist reservation strategy to ensure absolute convergence of genetic algorithm.The local optimal solution is approximated to the global optimal solution by combining the dual-population niche strategy with the improved adaptive genetic algorithm evolution strategy.Experimental results show that the batch scheduling algorithm in this paper can achieve 100% scheduling success rate under the premise of absolute convergence.Compared with other scheduling algorithms,it can make the astronomical application cluster achieve the effect of overall load balancing more effectively.The automatic deployment method of distributed astronomical application clusters and the batch scheduling algorithm based on genetic algorithm in this paper effectively solve the problems of difficult deployment and inefficient use of computing resources in distributed astronomical application clusters,and have certain practicability.The related research also provides a useful idea for the multi-type astronomical applications to be deployed agilely in the container cloud environment at the same time,which has certain application and promotion value.