Research Of Vector Tile Construction Method Based On Hadoop

The rapid development of computer technology promotes the rapid collection,processing and application of geographic information spatial data.Massive spatial data has been widely used in Web and mobile terminals.Tile maps based on pyramid technology have made great success in applying spatial data to navigation,map display,location-based Internet services and other fields.The huge need puts higher demands on the display rendering and production of tile maps.Traditional raster tiles have some disadvantages,such as huge amount of data,non-customizable style,time-consuming data updating,etc.Therefore,efficient construction and query based on vector tiles is the current development trend and research direction.Hadoop is the mainstream enterprise-level big data processing architecture,which uses MapReduce programming model to.efficiently process and analyze various kinds of massive data,and uses HDFS to store massive data.Therefore,using Hadoop to solve the production and data update of massive vector tiles is a highly feasible scheme.Based on the basic principle of Web vector tile map and related tile partition models,this paper proposes the dense-sparse vector tile construction algorithm considering the spatial distribution of vector elements,and uses KD tree to construct and query spatial indexes of massive tiles.At the same time,based on the vector tile construction algorithm,a distributed vector tile structuring algorithm based on MapReduce is designed and implemented.In addition,the solution of data skew in parallel computing and the influence of various parameters on the time consumption of parallel algorithm are studied.Based on this,this paper has done the following work:(1)Aiming at the limitation of rendering and transferring vector tiles on the Web,a tile partition model accounting for the spatial distribution of vector elements is proposed.According to the binary tree idea of spatial index KD tree,the partition model of tiles is established,and the dense-sparse vector tiles are constructed and the four-dimensional KD tree index model is established.Compared with grid vector tiles,the experiment shows that compared with grid vector tiles,it has better tile data balance and shorter network transmission time.(2)Based on the theoretical basis of vector tiles,this paper studies the parallel construction of vector tiles.Aiming at the limitation of traditional vector data,the massive spatial data model based on GeoCSV is realized,and the conversion algorithm from Shapefile to GeoCSV is realized.According to the MapReduce parallel construction task decomposition,the parallel construction of vector tiles is divided into three phases:Map,Combiner and Reducer,and MapReduce distributed data processing is carried out.(3)In order to solve the problem of data skew,a method of sampling data in Reducer phase is realized.An optimization algorithm is proposed to reduce the number of operations of Map phase function by using the minimum outsourcing rectangle of spatial data,which can improve the speed of batch construction of vector tiles.