Research On Deep Learning Based Spectral Reconstruction Methods Of Hyperspectral Images

While imaging the spatial structure of the target scene,hyperspectral images(HSIs)can cover tens or even hundreds of narrow-bands continuous spectra for each spatial pixel,which have the characteristics of map-spectrum integration and high spectral resolution.Profiting from the rich spectral information,HSIs can achieve fine distinction of different objects through their spectral characteristics.Therefore,HSIs have important application value in various fields,including geological exploration,environmental perception,agricultural remote sensing,and military operations.However,traditional hyperspectral devices utilize twodimensional sensors to scan in the spatial or spectral dimension to acquire three-dimensional data,so that the spatial resolution and spectral resolution of HSIs need to be compromised.Moreover,the scanning-based method requires a large amount of shooting time.Snapshot hyperspectral imagers can speed up the acquisition of HSIs,but at the expense of certain spectral resolution and spatial resolution.Whether it is a traditional hyperspectral device or a snapshot hyperspectral imager,the complex composition of hyperspectral cameras requires a large number of professional hardware parts and the equipment is expensive,which further affects the application of HSIs.To achieve low-cost and fast acquisition of HSIs with high spatial resolution and high spectral resolution,spectral reconstruction,that is,directly reconstructing corresponding HSIs from RGB images,has become a new research trend.Especially with the development of deep learning,spectral reconstruction methods based on deep neural networks have received extensive attention and research.Since the spectral recovery task is to estimate the missing band information of the HSIs through RGB inputs,it is a very complex nonlinear process,which easily leads to low accuracy of the reconstructed spectrum.Besides,spectral reconstruction realizes the mapping from low-dimensional space to high-dimensional one,and there is a one-to-many problem in the solution set,which makes it difficult to guarantee the reliability of the restored spectrum.In addition,the existing spectral reconstruction methods are of high complexity and cannot be well applied to low-power and lightweight computing devices.In order to deal with the above difficult problems,this paper conducts in-depth research on the spectral recovery methods of HSIs based on deep learning.Through statistically analyzing the data characteristics of HSIs,deepening the extraction of spatial-spectral features and utilization of prior information,and further exploring the lightweight design of spectral reconstruction algorithms,the rapid acquisition of high-quality HSIs is achieved.The main contributions are as follows:(1)Research on Adaptive Weighted Attention Network Model with Camera Response Prior:Aiming at the problem that most existing methods do not fully consider the different importance among the intermediate feature layers and ignore the utilization of the camera response function prior,resulting in low accuracy of reconstructed spectrum,this paper proposes an adaptive weighted attention network with camera response prior(AWAN-CRP).The backbone network forms dual residual learning through long and short skip connections,which promotes the stability and convergence of the model training.To model the interdependence among intermediate channels,an adaptive weighted-pooling channel attention module is designed.In addition,the proposed AWAN-CRP constructs a uniform-windowing secondorder non-local module to capture long-distance spatial contextual information and enhance the learning ability of the model.By embedding the camera response prior curve into the loss function to form a closed-loop constraint of the reconstructed spectrum,the accuracy and confidence of the predicted spectrum are further improved.In the NTIRE2020 Spectral Reconstruction Challenge Competition,the proposed approach ranked first in the accuracy index.(2)Research on Deep Residual Codec Network with Dual-domain Relational Augmentation:Aiming at the problem that there may be spectral aliasing phenomenon at the edges of HSIs,which can cause large errors in the high-frequency details of the reconstructed spectrum,and make the overall spectral reconstruction effect poor,this paper proposes a deep residual codec network with dual-domain relational augmentation(DRCN-DRA).Through extracting significant edge positions from the output spectral signals and the ground HSIs respectively,the model constructs a constraint on edge detail perception to retain rich high-frequency detail information in the recovered spectrum and improve the reliability of reconstruction results.Furthermore,the presented DRCN-DRA designs trainable channel-domain and spatialdomain relational augmentation modules to automatically emphasize informative features in spectral channels and spatial locations.The effectiveness of the proposed constraint and relational augmentation module is proved by ablation experiments.Under multiple public datasets,compared with other existing methods,the DRCN-DRA model contains obvious advantages in both objective quantitative indicators and subjective visual evaluation.(3)Research on Multiple Residual Deep Network with Multi-source Prior Information: To deepen the extraction of spatial-spectral features,make full use of prior information and further solve the problems of low accuracy and difficult reliability of reconstructed spectra,this paper proposes a multiple residual deep network with multi-source prior information(MRDN-MPI).First,the model takes the multi-residual connection structure as the core framework,which fully facilitates the transmission and utilization of low-frequency context priors from RGB images.Second,through introducing the semantic prior information of RGB inputs,the proposed MRDN-MPI designs a semantic-prior-driven feature aggregation module to optimize the quality of estimated spectra.Furthermore,by adopting more discriminative vectors instead of single digits as deep feature descriptors,the MRDN-MPI model makes the learned mapping function more robust and smoother.To obtain predicted spectra that are more consistent with the ground truths,the presented approach constructs a hyperspectral statistical prior consistency loss function to constrain the covariance relationship of the restored spectral bands.Finally,the effectiveness of the output HSIs is verified on the publicly available spectral reconstruction datasets.(4)Research on Proximity-spectral Interaction Lightweight Network with Structural Reparameterization: Aiming at the problem that the existing spectral restoration algorithms are high-complexity,which limits their application in low-power and lightweight computing devices,this paper proposes a proximity-spectral interaction lightweight network with structural reparameterization(Rep PILN).By constructing a network architecture stacked by parallel multi-way and multi-modal convolution units in the training phase,the deep spatial and spectral feature correlations can be effectively learned.With the help of structural reparameterization technology,the trained-finished network can be equivalently converted into an efficient and lightweight model in the inference stage,improving the running efficiency of the algorithm.In addition,to effectively improve the accuracy of the recovered spectra with a small computational cost,combined with the principle of hyperspectral imaging,the presented Rep PILN develops a coordinate-preserving neighbourhood-spectral interaction structure,which selectively highlights the features that are useful for improving the reconstruction performance.Considering that the spectral response curve is the core basis for distinguishing the attribute categories of ground objects,the spectral variation consistency constraint is designed to maintain the spectral continuity of the reconstructed hyperspectral signals.Finally,the proposed method achieves a desired balance between the restoration quality and model complexity.In conclusion,the research effectively addresses the problems of limited accuracy,low reliability and high algorithm complexity on spectral reconstruction of HSIs,and achieves the acquisition of high-quality HSIs at low cost and fast.The achievements can provide technical support for spectral reconstruction in many important fields such as aerospace,resource exploration,environmental monitoring and urban planning,etc.