Research On The Key Technology Of Image Emotion Semantic Analysis Based On Deep Learning

Emotion cognition is a kind of advanced cognitive activity of human beings.It should also be an indispensable part of artificial intelligence.The machines are required to have the ability of emotion cognition in human-computer interaction,so that the machines can act as the same emotion responses from humans.Psychological studies have revealed that visual content(such as images and videos)can evoke a variety of emotion response of observers.With the development of the Internet,people are also becoming increasingly interested in expressing their thoughts and feelings by uploading images and videos to social media platforms,such as Weibo,We Chat and Twitter.The visual content can convey and influence humans’ emotions.This thesis focuses on image emotion semantic analysis,which has a wide range of needs and applications in many fields,such as public opinion monitoring.It is important and interesting research work to understand which emotions that images convey and how those emotions are expressed through images.Image emotion analysis has become a hot topic in computer vision.Image emotion analysis refers to analysis the emotional semantics of image conveying and recognize the emotions induced by image.The goal of image emotion analysis is to establish the mapping relationship between image and emotion,which is a very challenging work due to the ambiguity of emotion semantic.The emotion cognition is highly subjective,which is related to culture background,personality and social environment.It is difficult to annotate emotion for images.Secondly,not all regions(pixels)of the whole image can evoke emotions.What regions or features in the image induce human emotion responses is an important issue in image emotion analysis.This thesis focuses on the key and challenging issues in image emotion analysis,that is,the emotion region detection,emotion semantic fuzziness and emotion dataset annotation,and explore effective methods to solve these problems.The main contents and contributions of this thesis are as followings:(1)This thesis proposed a cross-spatial pooling image emotion analysis method which focuses on the emotion regions in images.A forward emotion-oriented feature detection process is designed to detection the region related emotion,so as to realize the location of emotion regions.In this thesis,the fine-grained and pixel level emotion annotation is gained only based on image level annotation.The resulting emotion activation map can represent the regions which evoke emotion in the image,and can represent the contribution of each pixel to the evoked emotion.The proposed method solves the problem that the previous research works cannot explain how the convolution neural networks(CNNs)produce such classification result in image emotion analysis.The experimental results on Twitter Ⅰ and Emotion ROI datasets demonstrate that the proposed method can effectively learn the emotion representation features and improve the performance of emotion classification.(2)Considering the essential semantic characteristic of emotion categories,this thesis analyses the image emotion via deep metric learning.The analysis of image emotion is different from traditional image classification,because there is a far or near relationship between the emotion categories due to the fuzziness of semantic meaning of emotion categories.However,the existing image emotion analysis methods hardly take the category relationship into account.Thus,according to the current similarity of the sample pair,this thesis designs a progressive sampling and double weighting mechanism,in which more features that can do more contributions to updating model are learned from the informative sample pairs,accelerating model convergence.The experimental results on 8-class Art Photo and Abstract datasets demonstrate that the proposed model enables the deep metric learning to learn more discriminative deep embedded features,and the performances are better than the advanced classification methods.The ablation experiments prove that it is necessary to consider the relation of emotion categories for image emotion analysis.(3)The emotional annotation of emotion dataset is time consuming and costly.In order to address the problem of emotion dataset annotation on image emotion analysis.Based on the character of polarity and order,this thesis uses domain adaptive method to adjust the source domain emotion dataset with known labeled annotation to the target domain with no annotation by establishing the constraint of consistency,realizing the classification of unlabeled dataset in the target domain,that is,the weakly supervised emotion classification.This method is applied to Yunnan heavy color painting datasets,which first analyses the emotion of Yunnan heavy color painting datasets.The experimental results show that the proposed domain adaptive method is effective,and the proposed method of emotion distribution alignment is more suitable and effective for image emotion analysis.