Research On Parallel Processing Technology And Platform Of Hyperspectral Remote Sensing Mineral Data

Mineral resources are the important non-renewable resources during the process of social development.The hyperspectral remote sensing technology is widely applying in the field of surveying ground objects and it is of great help to mineral exploration and composition analysis.Hyperspectral remote sensing data has the characteristics of containing abundant information,large amount of data and complex calculation.The common stand-alone processing method can not meet its computation requirement.Efficient processing of hyperspectral remote sensing data by parallelization is the key to fully explore mineral resources.In order to improve the efficiency of hyperspectral remote sensing data processing,the main contents of this paper are as follows:(1)Designed a general efficient computing framework(HRSEGF)for hyperspectral remote sensing data processing.The framework uses the directed acyclic graph(DAG)to represent the set of tasks to be executed,and optimizes the process of mineral data processing from two aspects of subtask parallel and task node parallel.The subtask parallel computing method is used to reduce the processing time of hyperspectral data.And the reasonable scheduling algorithm is used to optimize the cloud computing resource allocation strategy in the task process.(2)For the subtask parallelism in HRSEGF,this paper builds a parallel optimization algorithm of hyperspectral remote sensing mineral data information extraction.The algorithm includes hyperspectral data dimensionality reduction,pure pixel extraction and spectral matching steps.The parallel optimization is carried out from the data storage mode,matrix calculation mode and the connection mode between steps.Through the experimental verification under different amount of data and different degree of parallelism,it shows that the parallel optimization algorithm can significantly improve the efficiency of task execution on the basis of ensuring the correct results of information extraction.(3)In order to implement the mineral task set in HRSEGF framework efficiently and use computing resources reasonably,this paper proposes a multi-objective scheduling algorithm based on hybrid particle swarm optimization.Through in-depth analysis of the heuristic algorithm optimization strategy,this paper adds a pre-generated search particle strategy to the hybrid particle swarm optimization scheduling algorithm.And combined with particle swarm optimization and artificial fish swarm algorithm,a new search method of hybrid particle swarm optimization is designed.By the experimental verification under different data sets,different degree of parallelism and different amount of computing resources,hybrid particle swarm optimization algorithm can effectively reduce the task set's execution time,improve the load balance and fit the HRSEGF computing framework in the field of mineral tasks scheduling.(4)Based on Hadoop + spark + HRSEGF framework,hyperspectral remote sensing mineral data processing platform is built.The platform adopts B/S architecture and looselycoupled design pattern,which has good versatility and scalability.The functional modules of the platform include computing resource management,parallel algorithm management,scheduling algorithm management and task flow construction.Users can choose the parallel processing algorithm component of hyperspectral remote sensing mineral data to build the whole DAG task flow of remote sensing data processing.And selecting the scheduling algorithm in the platform to optimize the execution process according to the actual needs.Mineral data processing platform meets the requirements of good human-computer interaction,users can efficiently process hyperspectral remote sensing mineral data without mastering the relevant knowledge of Hadoop and Spark.