Research On Distributed Hybrid Computing Technologies For Massive Remote Sensing Data

Remote sensing data is most important foundation resources of the scientific studyand production work in geo-information science and other related field, and itsapplication has already covered all related fields of geo-information science. Theprocessing, analyzing, computation and mining of remote sensing (especially highresolution remote sensing images), as one of its main application, is the commontechnical approaches of the study and production work for intelligent city, environmentmonitoring, geo-hazard early warning, land usage monitoring, vegetation monitoring,water and ocean remote sensing, agriculture remote sensing and atmosphere study.With the development of3S technology and information technology, the volume of themulti-source and multi-time and spatial remote sensing data is growing dramatically,sometimes the accumulated amounts are several TB or PB. Take Cheng E2images asan example, the volume of the original bands are add up to3TB, and the volume of theprocessed data is800GB. The processing of the image data which has a tremendousvolume, requires high computation ability, large storage capacity, which has beenbeyond the performance of common graphic workstations. For instance, on March2014, the Flight MH370of Malaysia airline company lost contact with air trafficcontrol. Although ten satellites are used in the rescue, the actual effect is not good, andthe reason is the complexity of massive data computation. The data obtained fromsatellites are firstly corrected before putting into use. Finding aircraft images in oceanremote sensing data calls for feature extraction and recognition with high precision.Current technologies cannot implement data processing, extraction and recognitionwith short time and large data volume; therefore lead to the unexpected situation.This article, based on the former study of the “data-oriented architecture” of ourresearch team, combined with the problem of data management and remote sensingdata computation, designed an organization model which can apply to distributedcomputation of remote sensing data. On the basis of the model, it discussed thedistributed computation model for remote sensing data. Deeply studying the big datatheory in geo-information science, in order to realize distributed hybrid computationand establishing the “all is data” big data thinking pattern. Focus on this point, this thesis put forward a generalized data-orientated system construct method. This method,through a flat data registration center, realized the unified management of thecomputation, storage, data and network resource of the cluster. On top of this, designedand implemented the distributed hybrid computation framework for massive remotesensing data, which solved the problems of theory study, data organization, multi-typecomputation combine and cluster resource management of massive remote sensing dataand implemented the distributed hybrid computation of massive remote sensing data.The innovation points of this thesis include:(1) Proposed a set of distributed computation theory and related techniquesfor remote sensing data. Based on the deep research of batch processing,stream-oriented computation, iterative computation, and other distributed computationmodels, put forward a divide method for distributed computation of remote sensingdata, which divides the computation into3processes: Split, Process and Reduce.Different computation models are implemented through different combinations of thesethree processes. At the same time, this thesis also designed a management model ofremote sensing data–elastic image pyramid model, which is used to solve the datamanagement problems in the process of distributed computation.(2) Proposed a big data viewpoint of “all things are data”. Based on theconception of “with data as the core”, aiming to realize “data orientation”, put forward abig data viewpoint of “all things are data”. This viewpoint incorporates the systeminfrastructure and data resources into the concept of “generalized data”, and studies itscategory system and description methods. On top of that established the unifieddescription method of infrastructures and data resources in distributed computationclusters through “generalized data”. Combining this method with “data-orientedarchitecture”, this thesis also established the “generalized data oriented” systemconstruction method.(3) Proposed a distributed architecture of data registration center and itsrelated techniques. Analyzed the architecture of centralized data registration centerand its potential problems, and based on the current metadata management model andtechniques of big data systems, this thesis put forward a distributed memory abstraction,and designed a distributed metadata management model–self-adaptive master-slavemodel. Finally, based on all the research achievements mentioned above, designed andimplemented a flat distributed architecture of data registration center.(4) Proposed distributed hybrid computation model for massive remotesensing data–“Chaos computation framework”. Through the deep researching of fundamental architecture of distributed computation framework, and the conception of“all things are data”, combined the distributed computation models of remote sensingdata, and making use of the “generalized data-oriented” system construction method,this thesis designed and implemented a distributed hybrid computation framework formassive remote sensing data–“Chao computation framework”. The application oflunar remote sensing image processing of “Cheng E2” indicates that Chaos has theability to execute distributed hybrid computation to large scale remote sensing data.The main theory and technical achievements of this thesis are:(1) This thesis improved the theory and technical framework of DOA andextended its conception of “with data as the core”. Also, this thesis, based on thetheory of “generalized data”, enhanced DOA.(2) Preliminarily established a set of distributed computation theory andtechnical architecture for remote sensing data. Proposed a divide method fordistributed computation of remote sensing data, which divide the whole procedure intothree processes: Split, Process and Reduce, and through the different combination ofthe three processes realized different computation models for remote sensing data.Meanwhile, this thesis also designed a management model for remote sensing data–elastic image pyramid model, which is used to solve the data management problems inthe process of distributed computation.(3) Preliminarily established a big data theory–“all thing are data”. Thisthesis defined the concept of “generalized data”,“traditional data” and “extended data”,and with reference to “Linnaean classification”, this thesis established the classificationand description method of “generalized data”. Finally, based on all these research work,realized the preliminary establishment of “all things are data” theory.(4) Designed and implemented the distributed hybrid computationframework for massive remote sensing data. Based on the viewpoint of “all thingsare data”, combined with the distributed computation model of remote sensing data,and making use of the “generalize data-oriented” system construction method, thisthesis designed and implemented the distributed hybrid computation framework–Chaos computation framework-for massive remote sensing data. This frameworkconsists of five layers, and can be referred to as the implementation of DOA.