| In applications of big data analysis,structured big data analysis and processing based on the table model is still the most basic requirement in many industries.As a table programming model,DataFrame has been widely used since it has a good abstraction of the data analysis and statistics process,and it is easy to use in programming.Although some DataFrame programming frameworks such as Pandas and Spark DataFrame have appeared,they still have many shortcomings.This paper makes a research focusing on the DataFrame big data programming model and framework from four aspects:programming model and framework,parallelism of operators,execution performance optimization,and cross-platform scheduling and computation.In this paper,a cross-platform unified DataFrame big data programming system has been designed and implemented.The primary work and contributions of this paper include:(1)This paper proposes a cross-platform unified DataFrame big data programming model and framework,and establishes a platform-independent DateFrame high-level abstraction,which provides good usability and cross-platform features for upper-level users.(2)Based on the Spark platform,this paper designs a DataFrame framework and proposes a lightweight global index and its construction method.Furthermore,the parallel construction methods of core DataFrame operators are proposed.(3)The large-scale DataFrame performance optimization methods are proposed based on the Spark platform,including an optimization method for lightweight global index construction,a optimization method for label slice query and update based on secondary index,and an operator execution performance optimization based on local index.(4)For different DataFrame operators and different data size in the cross-platform computing environment,an execution time evaluation model is proposed,which comprehensively considers the overhead of execution time and data transmission time between platform switching.Based on the model,a cross-platform DataFrame automatic optimized scheduling and execution method is implemented.(5)Based on the research of the above key technologies,a cross-platform unified DataFrame prototype system called Octopus-DataFrame is designed and implemented.The system provides complete DataFrame programming interface,and integrates several existing mainstream DataFrame platforms.Also,the system supports automatic platform selection and optimized scheduling across different platforms under different operators and data size.The results of experiment show that the proposed technologies and system methods have significant optimization effects for execution performance. |
