Explainability Methods Of Deep Learning Models For Optical Remote Sensing Image Classification

As deep learning technology gradually penetrates into various fields of social life,the ”black box” characteristics of deep learning models have aroused concerns about model trust and judicial ethics.The explainability methods of deep learning models aim to provide humanunderstandable decision logic inherent in the models and promote users to better trust and manage the models.Optical remote sensing image classification,as a basic task of remote sensing technology,has achieved remarkable development in recent years with deep learning technology.Since the task of remote sensing image classification involves many military and livelihood fields,the research on the explainability of deep learning models is also regarded as an important part of the future development of remote sensing field.However,due to the complex information and mixed semantic features contained in remote sensing images,it brings challenges to the research of deep learning model explainability for optical remote sensing image classification.Therefore,how to design effective explainable algorithms for the data characteristics of optical remote sensing images and the intrinsic mechanism of the models is a critical problem to be solved in remote sensing image interpretation and application.In this paper,we conduct research on explainability methods for deep learning models applied to optical remote sensing image classification tasks,and address the key problems of current research such as difficulty in constructing intrinsically explainable learning models,poor local visual explanation and lack of practical applications,and consider three perspectives from constructing network models with explainability,post hoc explainability methods based on saliency map and application algorithms based on explainability methods.The effective ideas of explainability algorithms based on natural optical images are taking into account.We focus on the characteristics of optical remote sensing images with rich low-level semantics,variable target scales and uneven target distribution,combined with deep learning model structure mechanism.And we aim to provide explainability for the decision-making of deep learning models and improve the understanding effect of model post hoc explanation.The main work and innovative contributions of this paper are summarized as follows.(1)An explainable network model based on fused prototype learning is proposed to address the problem that the prototype-based explainable network method cannot effectively extract targets of varying scales in optical remote sensing images and that the inference process after adding prototypes can cause degradation in network classification performance.The method learns image explainable features explicitly in the process of deep model learning and uses representative image patch prototypes and similarity scores to predict the output.Due to the rich low-level semantics and uneven target distribution of remote sensing image scenes,the general prototype-based interpretable network method cannot effectively extract and learn the prototypes in remote sensing image scenes.An improved high-resolution network module is proposed,and two classification heads,multi-polar feature aggregation head and hierarchical feature regularization head,are proposed at the output side,so as to make full use of features of different semantic levels and different scales to ensure that the prototype representation can be learned to match the remote sensing image.A weighted prototype learning loss function is proposed for the different scale features output by the hierarchical feature regularization head,so that the shallow feature information in the remote sensing image scene can be captured by the prototype and further enhance the explainability of the model.In addition,due to the different emphasis of fused features and hierarchical regularized features on classification performance and prototype learning,respectively,a regularization factor is proposed to balance the relationship between the two in the final loss.Sufficient experimental results demonstrate that the model exhibits good performance in terms of both classification accuracy and explainability.(2)To address the problem that most of the existing saliency map algorithms focus only on the end convolutional layers of the network model when interpreting predictions,which cannot produce convincing visual results for some scenes in optical remote sensing images,a probe weighted masking visual explanation framework applicable to remote sensing images is proposed.First,a probe-network-based class activation map method is proposed,and the algorithm can generate accurate visual explanation results on each convolutional layer of the model to be explained,and the probing of the explanation effect of each layer can be achieved for scenes with variable scale of discriminative targets and similar to texture images in remote sensing images.Subsequently,a weighted masking probability selection strategy is proposed to find the optimal interpretation layers for different scene classes.The strategy obtains the predicted values by continuously masking the regions with different scores in the saliency map and inputting them into the original model,then the predicted values of each masked image are weighted according to the scores to obtain the quantitative metric to evaluate the explanation effect of each layer of the saliency map for different categories of scenes,so as to finally complete the selection of the optimal explanation layer.The effectiveness of this method for post-hoc local explanation of the deep learning model of optical remote sensing images is demonstrated from the experimental results of both the fidelity and explainability of the explanation results to the model.(3)To address the problems that general saliency map algorithms cannot capture and reflect the rich low-level semantic information in remote sensing images and cannot generate effective saliency maps on shallow convolutional layers.Although the probe weighted masking method can reflect the shallow feature information on some scene categories,it only focuses on the feature patterns of a particular layer,therefore,a visual explanation method of optical remote sensing images based on channel saliency and saliency map fusion strategy is further proposed.The strategy of fusing the saliency map generated by the shallow layer and the end convolution layer is proposed,so that the saliency map providing local interpretation reflects more detailed spatial features in the original localization information and makes the final generated saliency map more consistent with the visual understanding of optical remote sensing images.To address the ineffectiveness of traditional algorithms in generating shallow saliency maps,we propose channel significance and use it as the importance weight of neurons in generating saliency maps to solve the computational problem of shallow saliency maps.And we propose to combine channel saliency with gradient weighting of positive bias derivatives for saliency map calculation in the end convolutional layer to ensure the accuracy of discriminative target localization information.The experimental results demonstrate that the method provides more applicable saliency map explanation results for optical remote sensing images,and the fidelity of the explanation results to the model is effectively improved.(4)Aiming at the specific application of explanation algorithms in optical remote sensing image classification tasks,an explainable convolutional neural network-based pruning method for remote sensing image classification networks is proposed by combining the existing network pruning algorithms in which the deep network channels are difficult to prune and the semantic meanings of the channels are ignored.Firstly,the improved explainable convolutional neural network method is proposed to adjust the training of the network model before compression,add explainable losses to the channels of the end convolutional layer according to the predefined pruning ratio,so that its part can learn the explicit semantic representation,and remove the non-explainable channels during pruning.To address the problem of how the heuristic pruning algorithm can effectively select the pruned channels of the other layers,a maximum response selection strategy based on the sensitivity function is proposed to rank the channels of each layer by calculating the absolute value of the sum of the kernel weight coefficients,and subsequently calculate the sensitivity of each layer using the accuracy loss from the pruning operation,and adaptively correct the pruning ratio of each layer according to the sensitivity.Finally,the performance loss caused by removing the channel parameters is regained by fine-tuning the compressed model.The experimental results demonstrate the effectiveness of this method for network model compression and enhance the explainability of the compressed model on optical remote sensing images.