Research On Key Technologies Of High-resolution Hyperspectral Intelligent Fusion Imaging

Hyperspectral imaging is an emerging technology that can capture dense spectral information of observed materials from visible bands to infrared and even the full spectral range to achieve accurate classification and recognition.It has been widely used in environmental monitoring,military reconnaissance,mineral exploration,smart agriculture,medical disease diagnosis,and other fields.Differing from RGB imaging,hyperspectral imaging integrates over a narrow spectral bandwidth to obtain continuous dense spectral information.To ensure the signal-to-noise ratio of the acquired hyperspectral image,under the same light energy,a single pixel needs a larger response region,which leads to the inherent contradiction between spatial and spectral resolution of acquired hyperspectral images,that is,the higher the spectral resolution of the image,the lower the spatial resolution.This contradiction greatly limits the popularization and application of hyperspectral imaging technology.Compared with hardware breakthroughs,a simpler and more effective way to improve the spatial resolution of hyperspectral images is introducing multispectral or RGB images with a higher spatial resolution of the same scene,and achieving this by accurate spatio-spectral image fusion imaging.This thesis deeply analyzes the research status and deficiencies of hyperspectral fusion imaging and proposes new fusion-based imaging architecture and methods with the cutting-edge deep-learning(DL)technology,which aims at the three major problems in hyperspectral fusion imaging:image registration,degradation analysis,and precise image fusion.The advanced methods proposed in this thesis are verified on plenty of remote sensing and natural hyperspectral image datasets.We further analyze the limitations of spatio-spectral fusion imaging,discuss the new trend of spectral super-resolution fusion and realize its hardware imaging systems.The details are as follows:(1)Aiming at the unknown degradation relationships between panchromatic and multispectral images in real scenes,an unsupervised blur kernel estimation method based on DL is proposed in this thesis.Firstly,the spatial and spectral degradation relationships of the panchromatic image and multispectral image are analyzed,and then a lightweight blur kernel learning network is constructed to simulate the degradation process.The spatial degradation process is realized by a strided convolutional layer,and the spectral degradation process is realized by a fully-connected layer.The linear interpolation operation is further introduced to make the spatial Gaussian blur kernel size-differentiable.By sending the observed panchromatic and multispectral images into the network,accurate spatial and spectral blur kernels can be learned.We further use the estimated spatial blur kernel to blur and downsample the original panchromatic and multispectral images.In this way,the original multispectral images can be used as training labels for the subsequent image fusion network,while the downsampled images can be used as network input.The real experimental results show that the blur kernels estimated by the proposed method can effectively improve the fusion accuracy of existing DL-based methods,and verify the effectiveness and practicability of the proposed method in the fusion of panchromatic and multispectral images in real applications.(2)To solve the key problems in multi-satellite hyperspectral image fusion such as unclear relationships of degradation,difficulty in image registration,and the phenomenon of spectral drifts.This thesis establishes new degradation models which are suitable for multi-satellite hyperspectral image fusion.Based on the new models,a joint optimization method of image registration field and blur kernel estimation between observed images is proposed,focusing on solving the problems of accurate registration and spectral drifts between multi-satellite images.The degradation models of the classical hyperspectral image fusion obey a rule that the high-resolution multispectral image and the low-resolution hyperspectral image are degraded from the same high-resolution hyperspectral image.However,when the two observed images are from different satellites,this theory no longer holds.We consider that in multi-satellite image fusion,these two images are obtained by degrading different high-resolution hyperspectral images,and propose a band-wise linear fitting model to simplify the problem.Further,the proposed method utilizes an image registration sub-network to predict spatial offset field of low-resolution hyperspectral images and uses bilinear interpolation to resample the hyperspectral images to match the high-resolution multispectral images.The registered images are then input into a blur kernel learning sub-network to estimate the spatial and spectral blur kernels,which consist of three layers,namely,a band-wise linear fitting layer,a fully-connected layer with weights as the spectral blur kernel,and a depthwise-convolution layer with the sizedifferentiable Gaussian kernel.We concatenate the image registration network and the blur kernel learning sub-network and train them jointly.After completing the registration and blur kernel estimation,a fast fusion sub-network is further introduced to realize hyperspectral image fusion based on a zero-shot learning strategy.The real experimental results show that the proposed method can effectively achieve multi-satellite hyperspectral image fusion imaging,which greatly reduces the fusion error caused by inaccurate registration and blur kernel estimation.At the same time,this method bridges the gaps between theoretical research and practical application of the hyperspectral image fusion problem,hence most of DL-based hyperspectral image fusion methods can be extended to multi-satellite fusion tasks based on the proposed framework.(3)Because the existing DL-based hyperspectral image fusion networks cannot effectively utilize the known spatial and spectral blur kernels,this thesis converts the hyperspectral image fusion problem into a high-resolution multispectral image guided hyperspectral image superresolution problem,and proposes a detail-boosting super-resolution hyperspectral image fusion network.The proposed method respectively constructs an image super-resolution module and a detail-boosting module.The image super-resolution module stacks the basic modules of classical natural image super-resolution networks,which aims to recover high-frequency details lying on high-resolution hyperspectral images.Further,the proposed detail-boosting module can effectively utilize the known or estimated blur kernels and the additional high-resolution multispectral images to further enhance the details recovered by the image super-resolution modules.By embedding the detail boosting modules into the original natural image super-resolution networks,we can transfer the classical image super-resolution network to hyperspectral image fusion.The experimental results on large-scale hyperspectral image datasets show that the proposed detail-enhanced super-resolution networks can obtain better hyperspectral image fusion results and has obvious advantages in visual effects and various image quality evaluation indicators.(4)Considering the problems of the difficult acquisition of hyperspectral images and expensive hardware in classical spatio-spectral fusion imaging,a spectral super-resolution fusion method is proposed based on multiple multispectral or RGB cameras.In this method,the lowresolution hyperspectral image in the above fusion task is replaced with a high-resolution multispectral image acquired by another RGB camera with different spectral response curves,and the spectral complementary information between two RGB images is utilized for accurate spectral super-resolution fusion imaging.Specifically,this method first generates training data on existing hyperspectral datasets based on the spectral response functions of two different RGB cameras,and utilizes it to train a spectral super-resolution fusion network.Then the RGB images of the same scene acquired by the two RGB cameras are sent to the trained network for reconstructing the high-resolution hyperspectral image.This thesis further analyzes the effects of various methods of acquiring another RGB image on the performance of fusion imaging,and builds an imaging hardware system.We also complete the preliminary imaging experiments and prove the consistency between the reconstructed spectrum and the real reference spectrum.