Research On Aspect-oriented Multimodal Sentiment Analysis Method

The emergence of various internet information exchange platforms has provided a lot of convenience for people’s lives.More and more online users express their opinions and emotions on social media,thus generating multimodal data in various forms such as text,images,audio and video.User-generated multimodal social media content often contains complex emotional information,and different aspects may express emotions of different polarities.Aspect-level multimodal sentiment analysis has practical applications in the fields of opinion analysis,sentiment dialogue,and especially goods and services review analysis.However,traditional sentiment analysis is coarse-grained,which focuses on the overall sentiment.Therefore,in order to more sufficiently reveal the fine-grained sentiment of social users,this paper focuses on text and image content in social media to perform aspect-level multimodal sentiment analysis,which aims to infer the sentiment polarity of target aspects in sentences by combining multimodal content.Due to the heterogeneous nature of multimodal data,aspect-level multimodal sentiment analysis is challenging in understanding multimodal data.This paper addresses the issues of aspect-oriented multimodal data representation,cross-modality data alignment and fusion,the cross-modal semantic gap,to enhance the performance of aspect-level multimodal sentiment analysis by enhancing the semantic understanding of aspects in multimodal data and improving the alignment of aspects and multimodal contents,ultimately.The main study of this paper are as follows:(1)Most of the current multimodal sentiment analysis methods focus on the mining and fusion of multimodal global features,ignoring the correlation of more fine-grained multimodal local features,which greatly limits the performance of aspect-level multimodal sentiment analysis.Therefore,this paper proposes a novel aspect-level multimodal sentiment analysis method based on global-local feature Fusion with co-Attention,which introduces gated local co-attention mechanism to compensate for the lack of fine-grained information in global text features and global image features based on the construction of aspect-guided multimodal global co-attention to achieve finer-grained multiple interaction alignment between text and images.Then,according to the characteristics of multimodal features at different levels,different fusion methods are designed to achieve the deep fusion of multimodal features at different levels.Finally,A series of comparisons of the proposed method with other related methods are conducted on two publicly available datasets,and experiments show that the method is able to improve the aspect-level multimodal sentiment analysis.(2)Most of the existing methods ignore the directional semantics of aspect words in intra-modality context and fine-grained alignment of inter-modality context,which greatly limits the performance of aspect-level multimodal sentiment analysis.In addition to tackling the problems mentioned,this paper proposes an aspect-level multimodal co-attention graph convolutional sentiment analysis model.The model uses the self-attention mechanism with orthogonal constraints to generate semantic graphs for each modality,based on which graph convolutional network is used to obtain aspect-oriented local semantic representations within the modality.And two different directions of gated local cross-modal interaction mechanisms are designed to recursively implement fine-grained cross-modal associative mutual alignment of textual semantic graph representation and visual semantic graph representation,thus reducing the heterogeneous gap between modalities.And the aspect mask is designed to select the node features related to each aspect in the modal graph representation as the emotional representation,and the cross-modality loss is introduced to reduce the difference of cross-modal aspect features.Finally,a comparison experiments with benchmark methods is performed on two public multimodal datasets,the experimental results show that the proposed method can significantly improve the performance of aspect-level sentiment analysis compared to the benchmark method.