Research On Multicore Job Support And Scheduling In JUNO Distributed Computing Environment

The Jiangmen neutrino experiment(JUNO)is the second large neutrino experiment project in China,which will produce the original data of the magnitude of PB every year.To meet the needs of massive data processing and analysis,the Institute of high energy physics of Chinese Academy of Sciences has been built JUNO distributed computing system by integrating the heterogeneous computing resources from the cooperative organizations all over the world.With the accumulation of experimental data and the increase of event complexity,the consumption of memory resources of JUNO task is further expanded.The existing single core processing mode of the experiment will be difficult to meet the memory requirements of each job.The JUNO experiment plans to use multicore jobs to alleviate this problem.The supporting and scheduling of multicore jobs has been studied in this paper.The main contents are as follows:(1)According to the need,supporting JUNO multicore parallel jobs is studied as an important component of JUNO distributed computing.Based on pilot agent,we got the heterogeneous resource allocation strategy.This strategy reduced the delay in matching between jobs and resources,and facilitated users to access heterogeneous resources.In order to realize the hybrid scheduling of single core and multicore jobs,studying the existing scheduling strategies,a job scheduling model of public pilot has been presented.The scheduling of jobs have been transferred from the site to the agent,which can alleviate the complexity of job scheduling in the site.(2)A job scheduling algorithm for job resource requirements has been proposed based on study multicore job scheduling.This algorithm combined backfill strategy and priority scheduling algorithm.In the presence of free resources,the priority of multicore jobs can reduce the production of resource debris.Then,the resource is reserved for the operation of the two level priority to reach the threshold,which has beeen alleviated the problem of hunger.The experimental results show that this algorithm has been improved the overall resource utilization and the efficiency of operation to a certain extent.(3)Considering the need of the JUNO experiment and users,the structure and performance of distributed computing system,and other factors,the overall architecture of the multicore model has been designed.Finally,the scheduling and execution of multicore jobs has been implemented using the workload management service of the DIRAC middleware and adopting new techniques,such as resource sharing pool,dynamic matching service and parallelization execution etc..The multicore job has been applied to the data processing and analysis.The pressure measurement results show that its performance meets the requirements of the JUNO experiment.