The Influence Of Emotional Context Information Of Audio-visual Channel On Ambiguous Expression Emotion Recognition

Facial expressions are important symbols for humans to convey emotional information,and their psychological states and behavioral responses in emotion recognition are one of the topics considered by scholars.In daily life,facial expressions do not appear in isolation,but always in emotional context information.Emotional context information can be internal information that accompanies facial expressions,including voice expressions and body expressions.External information that appears,including language,natural scenes,and smells.This information is a challenge for people to rule out their effects when recognizing the emotion of an expression.Some scholars have summarized the two effects of emotional context information on facial expression emotion recognition: One is the emotional-context effect,which refers to expressions with clear valence and high emotional intensity(such as happy,sad,fear,and angry expressions).When the context information and expression valence are same,it will promote people’s emotional recognition of expressions,and when the context information and expression valence are different,it will hinder people’s emotional recognition of expressions.The other is the emotion-evoked effect,which means that for expressions with unclear valence and low emotional intensity(such as neutral and surprised expressions),the context information will make people recognize the emotion of expressions in accordance with the context information valence.Looking back on previous studies,researchers have investigated the respective roles of emotional context information in visual,auditory,and olfactory channels on ambiguous expression emotion recognition,but it has not been found that in the ambiguous expression emotion recognition,the integration and difference of the emotional context information of each channel.Because behavior score can examine the emotional recognition of ambiguous expression from the valence dimension,eye tracking technology can examine the emotional recognition of ambiguous expression from the attention dimension.Based on this,this study uses the method of behavior combined with eye-movement to explore the channel effect of emotional context information on ambiguous expression emotion recognition from the valance and attention dimensions.From the perspective of audio-visual channel integration,Study 1 investigates the effect of dual-channel emotional context information on ambiguous expression emotion recognition.The emotional context information of visual channel selects emotional expressions(happiness and sadness),the emotional context information of auditory channel selects emotional sounds(laughter and crying),and the information of two channels is presented at the same time,in order to explore the influence of valence consistency of dual-channel emotional context information on ambiguous expression emotional recognition.Behavioral experiments show that for the emotional recognition of ambiguous expression,the emotional context information of audio-visual channel promote each other when the valance of information is same,and the emotional context information of audio-visual channel hinder each other when the valance of information is opposite;eye-movement experiments show that pupil diameter,duration of the first fixation and fixation time are more affected by emotional context information of the visual channel.Among them,the pupil diameter is influenced by the valence of visual emotional context information,which causes a negative attention bias for the ambiguous expression,and the other indicators show that people’s attention preference for the eyebrow and eye area of ambiguous expression is more affected by sad expression,and the attention preference for the mouth and nose area of ambiguous expression is more affected by happy expression.From the perspective of audio-visual channel comparison,Study 2 investigates the effect of single-channel emotional context information on ambiguous expression emotion recognition.The experimental materials are the same as study 1,but the visual and auditory emotional context information are presented in a single channel,in order to explore the strength and weakness of two information on emotional recognition of ambiguous expression.Behavioral experiments show that for the emotional recognition of ambiguous expression,auditory emotional context information has a greater impact than visual emotional context information;eye-movement experiments show that the pupil diameter is influenced by the valence of emotional context information and has a negative attention bias for the ambiguous expression,and the other indicators of the first fixation duration and fixation time are consistent with the behavior results,compared with the visual emotional context information,people have attention preferences for ambiguous expression under the condition of auditory emotional context information.In summary,this study proves from the valence and attention dimensions that there is a channel effect in the emotional context information affecting ambiguous expression emotional recognition.The main points are as follows:(1)Dual-channel emotional context information has a channel-integration effect on ambiguous expression emotion recognition,but visual emotional context information has a greater impact on the attention preference of ambiguous expression interest area.(2)Single-channel emotional context information has a channel-contrast effect on ambiguous expression emotion recognition,which shows that auditory emotional context information has a greater impact on ambiguous expression emotion recognition,but there is no clear attention preference on ambiguous expression interest area.