Image Sentiment Analysis Via Active Sample Refinement And Multi-Grain Semantics Mining

Image sentiment analysis is a research focus in the computer vision field.It has significant economic value and social value and contains very wide application prospects.For example,it helps to predict whether people have psychological problems and intervene in time,to predict the satisfaction of users about a certain type of product in the mind,to predict the fashion trend of a product,and to track the current public opinion trends.However,owing to the subjective differences of annotators,image sentiment datasets have some problems currently,such as fuzzy emotional semantic annotation and the lack of high-quality image samples.Meanwhile,on the implicit cross-modal semantics among heterogeneous features and the multi-grain semantics has not been fully explored.To address these problems,this paper proposes its research work on image sentiment analysis based on active sample refinement and multi-grain semantics mining.To alleviate the problem of lack of high-quality samples,it proposes a novel active sample refinement model.To accurately depict images’visual content,it deeply mines the cross-modal semantics among heterogeneous features.And the multi-grain semantic information is obtained through the affective region detection algorithm,which is complementary for image sentiment anlaysis.Hence,this paper includes several algorithms,such as active sample refinement,cross-modal semantics mining,multi-head data augmentation,and affective region detection.The main contents are as follows:(1)Image sentiment analysis via active sample refinement and cross-modal semantics mining:By borrowing the idea of active learning,the paper firstly designs an active sample refinement strategy to alleviate the two problems,such as the lack of high-quality samples and fuzzy sentiment annotations.It refines the image samples with high-quality to augment the original datasets.The the SIFT and VGG features are extracted in turn and the low-level discriminant correlations between the heterogeneous features are mined after active sample refinement.Finally,the cross-modal semantics is used to train classifiers for image sentiment analysis.Experiments show that the proposed model outperforms most mainstream baselines.Furthermore,this model only requires two types of features,which makes it easy to reproduce and deploy on computers.More importantly,the proposed ASR strategy can be combined with other data augmentation methods to make sample refinement and enrich the original dataset.Hence,it can address the data scaricity well and is a good complementarity to current augmentation methods.The proposed active sample refinement strategy has a certain generalization.(2)Image Sentiment Analysis via Active Sample Refinement and Cluster Correlation Mining:In this section,the paper modifies the model proposed in the last section.It firstly uses the latest SENet features with more powerful discriminative ability to replace the SIFT and VGG features,and then it uses cluster correlation mining to replace the discriminative correlation mining.And more valuable cross-modal semantics is obtained in turn.Experimental results show that the SENet features outperforms those traditional and deep learning features.Also,the cluster correlation mining can retain much more valuable discriminative information,making the cross-modal semantics more explanatory than before.The Experiment shows that active sample refinement and cluster correlation mining algorithms are effective and robust.They all contribute to improving the final accuracy.Especially in the fine-grained dataset,our model gets about 4.55%performance improvement compared with the ASRF~2(U).(3)Image sentiment analysis model based on multi-head data augmentation and multi-grain semantics mining:In order to alleviate the problem of the lack of high-quality samples further,this paper introduces the multi-head data augmentation mode.On the one hand,it combines the proposed active sample refinement strategy with the state-of-the-art automatic data augmentation method.This helps to promote both the“quality”and“quantity”of the original dataset.At the same time,the affective region detection model is introduced to detect local regions with very strong emotional semantics.The detected affective regions will be added into the original dataset to form a multi-grain image dataset,which can depict the sentimental semantics in images from divese perspectives.Finally,the well-known deep mutual learning model is introduced to fully mine the implicit knowledge among two heterogeneous SENet networks.Then fusing the heterogeneous SENet features adaptively for multi-grain image sentiment analysis.Experimental results show that the proposed model based on multi-head data augmentation and multi-grain semantics mining can further improve the final accuracy.The proposed model is superior to all the other baselines.For example,a1.07%performance improvement can be observed compared with the model proposed in the last section.The main contributions of the paper are as follows:(1)it proposes an active sample refinement strategy,and a novel multi-head data augmentation method combined with current mainstream data augment method and the ASR strategies.It can promote the original datset from the two perspectives of“quality”and“quantity”.(2)It mines the cross-modal semantics among the heterogeneous features through the cluster correlation mining algorithm,which helps to better characterize image sentimental content.(3)It uses the multi-grain images to complete image sentiment analysis complementarily from the global and local perspectives.