Research On Land And Resources Spatial Big Data Cleaning Method Driven By Rule Engine

With the development of information processing,the department of land and resources has accumulated large amount of spatial data with great value.Promoting the big data construction of Land and Resources and the resource data sharing between department of land and resources and other departments such as department of surveying and mapping has been the core of national policy and Land and resources' work.The unification of data standard is the foundation of data sharing.The land and resources data features as variety,velocity,veracity cause difficulties in data sharing.Data cleaning technology in the field of Data quality research can solve these problems.Among the tranditional data cleaning methods,some are hard coded and with less flexibility,the others base on the stand-alone mode and can't apply to big data cleaning.As for the widely used rule engine Drools,there is a tight coupling between it's cleaning rules and program codes and the Drools rule engine bases on the standalone calculation environment.In summary,thers is few studies focusing on integrating the big data technology and the rule engine.Based on the analysis result of the land and resources data characteristics and the land resources big data applications' requirements,we proposed a cleaning method for land and resources spatial big data driven by rule engine.On the basis of the data structure of the ArcSDE Geodatabase and the anylysis result of the SDELOB encrypt algorithm,firstly,this method desingned and implemented a spatial big data transformation algorithm with a distributed and parallel computing framework—MapReduce.This algorithm realized the spatial data transformation from RDBMS to big data environment of department of land and resources.With the support of transformed data,we came up with a cleaning rules' encoding schema to ensure that the cleaning rules separate from the program codes.Under the support of the rule engine's function modules including rule definition module,rule configuration and arrangement module,rule examination module,rule parsing module and rule execution module,it implemented a cleaning rule's dynamic extension and a flexible rule arrangement to ensure the rule's high reusability.The land and resources rule engine combined with the MapReduce to form a efficient and accurate spatial big data cleaning technology.It is the most appropriate way to solve the problem of land and resources spatial big data cleaning.The experiment result evaluates effectiveness of this method,and the result also proves that this method highly increased the efficiency of spatial data transforming and spatial data cleaning on the premise of ensuring the accuracy.