| The emotion is a highly generalized set of physiological,cognitive and behavioral reactions of human beings and is a key component of people’s emotional communication.Generally speaking,the information that emotions can express includes physiological signals such as EEG,respiratory heart rate and non-physiological signals such as expressions,speech and gestures,while non-physiological emotion data has received much attention due to its rich source,low acquisition cost and complete range of modalities.In addition,the research on single-modality emotion recognition suffers from the disadvantages of low recognition accuracy,poor stability and susceptibility to interference,and often fails to reflect the complete emotional expression.To this end,this paper focuses on multimodal emotion recognition of non-physiological signals based on video data from natural scenes,mainly involving three modalities:facial expressions,natural language and audio signals.At present,there are three research challenges in multimodal emotion recognition based on non-physiological signals:efficient representation,cross-modal fusion and unaligned co-learning.How to extract features for the specificity of different modalities to obtain a more effective and highlevel representation;How to fully consider the complementary information between modalities and ensure the integrity of the feature learning process,i.e.the fusion across the semantic gap of each modality;How to directly handle the unaligned asynchronous multimodal data and allow each modal data to learn together to achieve the reinforcement of feature information of different modalities.To address the above issues,the main work of this paper is as follows.Firstly,this paper proposes a Transformer-based bimodal fusion network for the audio and video modalities.The two core issues of reducing redundant features and enhancing complementary features are mainly considered in the fusion network.For reducing redundant features,inter-modal interaction is performed after intra-modal feature selection by self-attention learning.For boosting complementary features,feature reinforcement is achieved by weight multiplication.With the same unimodal representation,the network achieves 76.76%accuracy in the dataset RAVDESS compared to other fusion approaches.Secondly,the research of multimodal emotion recognition based on manual alignment is achieved by introducing text modality.In this paper,a non-homogeneous fusion network is proposed to achieve multimodal information interaction.The approach takes into account the difference in information density of different modalities,i.e.,visual and audio are lowlevel signal features,while text has high-level semantic features.Feature enhancement of visual and audio modalities is achieved by designing an attention fusion module with aggregation blocks,which overcomes the squared complexity operation caused by pairwise attention during the fusion process and improves the ability to effectively integrate complementary information among the modalities.The proposed model has been demonstrated by conducting extensive comparison experiments and ablation experiments with manually aligned settings in two publicly available datasets,CMU-MOSI and CMU-MOSEI,achieving FI values of 87.6%and 84.95%,respectively.Finally,this paper directly targets the unaligned multimodal sequences,addressing the problems of accurately distinguishing between modal consistency and modal specificity and multimodal learning training imbalance.To address the problem of co-learning of unaligned multimodal sequences,feature alignment is achieved using self-attention and crossmodal attention,with the loss of contrast learning for constrained alignment.To address the problem of unbalanced multimodal learning,the"inter-modal difference rate" metric is used,in which different modalities are dynamically assigned different gradient coefficients according to the difference ratio between the modalities during the training process.In this paper,experiments are conducted on two unaligned publicly available datasets and achieve F1 scores of 85.9%and 85.94%,respectively.The effectiveness of the proposed model is verified by qualitative and quantitative analyses of the comparative learning loss and modal difference rates. |
