Zero-shot Architectural Image Classification Based On Dual Attention Mechanisms And Weighted Graph Convolutional Neural Networks

Accurate architectural style classification is of great significance to the study of architectural history,the conservation of architectural heritage and urban construction.However,due to the similarity of different architectural styles and the difference of same architectural style,there is a lack of labeled architectural categories in the dataset.Therefore,using zero-shot classification technology to classify architectural images with missing label data has become a worthy topic to study.At present,there are two main shortcomings in the classification of architectural styles: First,due to the uniqueness of architectural styles,it is difficult to effectively extract the key features of architectural styles for most feature extraction methods.Second,a lot of emerging architecturals have early architectural elements,and the relationship between different categories of semantic labels is similar,so it is difficult to learn a classifier with higher fitness.Channel-spatial attention can focus on important areas related to tasks in the image and ignore unimportant elements.The graph convolutional network is able to use the knowledge graph to express the relationships between categories and update the feature vector of each node by aggregating the neighbourhood features of the nodes to generate a classifier relevant to the semantic data.To be able to effectively extract the main and detailed features of the building and find the strong correlation of each style’s semantic labels in the absence of architectural image label data,this paper studies zero-shot architectural image classification based on a dual-attention mechanism and weighted graph convolutional networks.The main work is as follows:(1)In order to effectively locate architectural elements related to classification tasks in architectural images,a zero-shot architectural image classification method based on dual attention mechanism is proposed.Firstly,two models of channel attention and spatial attention to enhance the representation of specific regions of the image are introduced.The channel attention network is used to detect different channel weights to locate the architecturals in the image;the spatial attention network is used to embed the location information into the channel attention map to capture the detailed features in the target to obtain feature representations with two dimensions of channel and space.Secondly,in order to reduce the information loss during spatial mapping,the generator is used to reconstruct the visual features.Finally,a zero-shot architectural image classification model embedded in public space is designed.The visual features and semantic features are aligned in the subspace,and the classification task is realized by the nearest neighbor matching.The effectiveness of the method is verified by experiments.(2)Considering the strong correlation between semantic labels of architectural styles,a zero-shot architectural image classification method based on weighted graph convolutional network is proposed by using explicit knowledge graph to mine the relationship between categories.Firstly,the feature extraction network is used to extract the architectural features into vector representation.Secondly,the hierarchical distance relationship between all style labels is used as a priori knowledge to construct a graph structure,and the distance between semantic features is calculated to weight the relationship between architectural style categories.The weighted graph convolution network is used to train and update the features of categories to alleviate the over-smoothing problem caused by too many network layers,to improve the aggregation of images of architectural in the same classes and enhance the distinction between different categories,and trains classifiers for all architectural style categories.Finally,the visual features are dot-producted with the classifier,and the classifier is used to classify the unknown styles in the prediction process to further enhance the transfer ability of the model.The experimental results show that the proposed method improves the average classification accuracy by0.6 percentage points on the Architecture Style Dataset compared with the state of the art zero-shot learning method.(3)A zero-shot architectural image classification system is implemented.A zero-shot architectural image classification system based on dual attention mechanism and weighted graph convolutional neural network is designed and implemented.