Research Of Acceleration Of Kylin Cubing Algorithm Based On FPGA

Massively parallel processing and columnar storage have increased SQL query spe-ed from hours to minutes,but this is far from meeting the real-time needs of interactive analytics.Apache Kylin is distributed online analytic processing based on the space-for-time idea.Kylin can provide multidimensional analytic capabilities and sub-second standard SQL queries for very large datasets.However,the increasing dimensionality of C-ube brings the problem of dimensional explosion,and puts tremendous pressure on dis-k and network transmission.The following work is carried out in this thesis to solve the problem of dimensional explosion when building multidimensional cubes in Kylin.Firstly,the Deflate hardware compression model based on FPGA(Field-Programm-able Gate Array)is proposed.The compression ratio of 2.33 and the compression speed of 2.87GB/S are obtained by hardware optimization means such as Area,Loop-Carried and loop expansion.The compression speed of this model has significantly improved th-e compression performance compared with the software implementation of Deflate algorithm.Then a heterogeneous distributed three-tier model design based on hardware compr-ession acceleration is proposed,and comparative experiments are conducted to verify that the heterogeneous clusters have significant improvements in read,write,and comput-ation performance.Finally,this thesis propose to build a distributed Cubing model based on heterogen eous acceleration at the query engine layer,and perform greedy-based dynamic pruning optimization on it.Based on the above work,comparison experiments are conducted between the Cub-ing model with heterogeneous acceleration and the Cubing model with dynamic prunin-g acceleration only.The experimental results show that the Cubing model with heterog-eneous acceleration obtains a 3.7 times speedup ratio in the construction time and a 2 ti-mes speedup ratio in the average query time.The Cubing model based on heterogeneou-s acceleration proposed in this thesis can accelerate the construction and querying of m ultidimensional cubes to relieve the IO pressure caused by dimensional explosion.