Research And Application Of Hyperspectral Imaging Acceleration Method Based On Spark Platform

In recent years,hyperspectral technology has been widely studied.Among them,hyperspectral image has been widely used in scientific research because of its advantages of both macro information and micro information.Compressed sensing proposed in recent years is a sampling law.It can sample signals that meet certain conditions at a frequency far lower than that required by Nyquist;then,using numerical optimization algorithms,it can accurately reconstruct the original signal from the sampled signal.The hyperspectral image signal is sampled by compressed sensing;the process of calculating the hyperspectral image data from the sampled data using numerical optimization algorithms is the hyperspectral imaging studied in this paper.Because the dimension of the matrix involved in the hyperspectral imaging process is very large,the optimization algorithm involved in the reconstruction of the original signal from the sampled signal in a single machine environment is very slow.To solve this problem,the distributed parallel computing research of hyperspectral imaging algorithms on Spark platform is carried out in this paper,in order to significantly improve the computing efficiency of hyperspectral imaging.In this paper,we first study the performance bottlenecks of hyperspectral imaging algorithms running on a single computer,and then propose a method to transfer hyperspectral imaging algorithm to Spark platform.The innovation of this method lies in: The sparse matrix in hyperspectral imaging algorithms is encoded by the COO format;the distribution rule of nonzero values of observation matrices in hyperspectral imaging algorithms is found;a method of constructing sparse matrix RDD quickly is designed;a distributed computing method is designed to compress the computation,which avoids the invalid computation;according to the nonzero value distribution law of sparse matrices,a load balancing partition is designed to avoid data skew and redundant shuffle operations.based on the previous methods,the serial hyperspectral imaging algorithm is transferred to Spark platform.Finally,the algorithm is tested comprehensively based on multiple sets of data.The test results show that the original signal reconstructed by the algorithm is consistent with regard to values with the original signal reconstructed by the original algorithm in the single machine environment;in addition,in the experimental environment of this paper,the execution efficiency of the algorithm is nearly 6 times higher than that of the original algorithm,Compared with MATLAB,the efficiency of parallel implementation increases with the growth of data scale.In addition,based on the proposed hyperspectral imaging algorithm under Spark platform,a hyperspectral image management system is designed.It shields the complex design behind the hyperspectral imaging system for the user,presents a simple interactive interface for the user,and enables the user to manage the hyperspectral data conveniently.Finally,the system is tested comprehensively.