| With the development of information science and technology, mass data emerges at the same time, according to the transmission and the calculation of mass data, computing ability and storage capacity of single node has become the bottleneck of data processing, more and more valuable data are not being used in single machine. In view of the super computing cluster has and cheap cost advantage, making the high performance of large-scale machine learning and scientific computing computing to make snap forward.At the same time, the modern graphics processing units and multi core architecture has become the universal parallel computing platform, it can greatly accelerate the application of science.Multiple GPU workstations that have a trillion times peak computing power can accelerate computing problem.Parallel computing framework is like a raging fire,a calculation method based on the traditional grid and hard disk, and the HDFS calculation and memory based on a popular now. Due to the limitation of the traditional parallel programming model,and also put forward higher requirements on the new parallel programming framework.Hadoop Mapreduce is one of the most popular open source distributed computing frameworks, which supports the above grade TB data processing, widely used in large clusters composed of thousands of commercial of the machine. However, the Mapreduce will repeatedly from the file system to read the same data, due to slower access disk I/O.It is the catalytic scalability requirements in saving cost and realize the system period, the concept of Spark emerges, and is considered to be the solution to the current large-scale data storage and processing the better scheme. USA AMP Laboratory of University of California at Berkeley developed the Spark framework, for the iterative algorithm or the interactive query repeatedly read and write the file system to the problem of low efficiency, through modeling and analysis on the behavior of memory, memory usage of decision automation and replacement strategy optimization.The main objective of this paper Spark MPI/GPU parallel processing mechanism is effective to improve the development efficiency of the distributed environment, and in terms of performance with the original implementation is quite. Not only for the current distributed parallel computing of MPI and high performance computing GPU tasks and resources how to allocate to achieve efficient and reasonable utilization of the problem.Through the development of environmental laboratory existing, discusses in the emerging development framework under the Spark framework, efficient and reasonable allocation of tasks and resources.In this paper,the development of architecture makes full use of the Spark cluster for task scheduling and resource management advantages,and the original MPI tasks and GPU tasks reasonable embedding in Spark.Spark MPI/GPU parallel processing mechanism of the process, the client puts the data on the HDFS on the Master node and distributes the data to each node.Spark submitted the Job task to Yarn, application program manager on the Yarn responsible for the management of the entire application, identification of MPI task or GPU task, Spark assigns MPI task or GPU task to each node further.Also calls the corresponding MPI program or the GPU program,each node timing report to Yarn this node resource usage and operation state.Finally, computing the results back to the Master node, the Master node upload to HDFS.The frame is made of parallel computing cluster framework and distributed Hadoop file system by Spark cluster resource manager Yarn, support the calculation of the distributed memory resident. This framework based on Hadoop platform, the GPU and MPI embedded Spark processing mechanism,realize the parallel computation of large-scale data, improve the data throughput. The test results show that, Spark MPI/GPU hybrid computing programming framework can significantly improve the processing speed, reduce the workload required for development. |
