Research And Implementation Of High-throughput Computing For High-performance Computing Systems

High-throughput computational applications such as gene sequencing and economic models such as MARS are increasingly important in modern scientific computing.However,due to the nature of high-performance systems,large-scale and loosely coupled applications can not be implemented directly on high-performance computing platforms.At the same time,the deployment and maintenance of high-performance computing systems and high-throughput computing systems will result in the waste of computing resources.In order to solve the problem of handling large scale and loosely coupled applications on high performance computing systems,this paper proposes to design and implement Teno,a framework for handling high throughput computing jobs on high performance computing systems.Do not need to modify the configuration of the Slurm system of the cluster resource manager,the hierarchical scheduling subsystem in Teno realizes hierarchical scheduling of computing resources by means of Slurm system.The outer layer uses Slurm system for scheduling in computing nodes.The inner layer uses the HTCTeno subsystem to kernel Dispatch for the unit.High-throughput computing subsystem in the traditional Master-Worker model for research and improvement,the "PUSH" mode to "POLL" mode,changing the way the task distribution,which greatly enhance the implementation efficiency of Teno system and can Scalability;The implementation of the engine subsystem to the task of forest organization and construction of operations,to solve the problem of describing large-scale operations;monitoring subsystem design,fault recovery and task retry mechanism to ensure the stability of high-throughput computing,Reliable execution.Through different experimental tests,it is proved that the task throughput of Teno system is 10 times that of HTCondor,the task execution efficiency is 55.08% higher than that of Slurm system,and the computing efficiency is improved by 43.12%compared with Slurm.When the task reaches the maximum scale of 10 million,12,000 The processors perform computational tasks with a run time of 16 seconds,and the effective utilization of computing resources can reach more than 90%.