Visual-auditory Cross-modal Emotional Conflict With Music As Background

Most researches about emotional conflict were done with visual or auditory stimuli separately. There are some studies concerned about the effect of enhancement between visual and auditory stimuli. Based on the previous studies, the current study concerns about what happens when there are visual channel emotional conflict and visual-auditory cross-modal emotional conflict simultaneously.In experiment1we manipulated the emotional attribute of current visual stimulus, the previous visual stimulus and the auditory stimulus, which are positive and negative; the interval between two visual stimuli, which is long and short. The pictures are selected from the International Affective Picture System (IAPS)(2008), which are classified as positive and negative. The music we used is Allegro-A little night music from Mozart as positive and Kol Nidrei form Bruch as negative, according to Mitterschiffthaler’s functional MRI study of happy and sad affective states induced by classical music (2007). Thirty nine participants from a university took part in the experiment. Then they are told to judge a series of pictures on the computer as positive or negative with no music, positive music and negative music as background. The reaction time and the accuracy are recorded. As all the accuracy is high enough, we analyze the difference of reaction time.The result shows that, when visual and auditory stimuli both exist, they take effect independently. The manipulation of visual stimuli can apparently educe visual modal emotional conflict. And the visual modal emotional conflict is related to the interval between the visual stimuli, which is to the current negative visual stimulus, it disappears when the interval is long, while to the current positive visual stimulus, it always exists whether the interval is short or long. To the visual-auditory cross-modal effect, as more attention is needed when we concern about visual-auditory cross-modal, there may be two kinds of effect in the current study, that is visual-auditory cross-modal emotional conflict and visual-auditory cross-modal negative effect.The second experiment is an ERP experiment, a preliminary study of the complex effect of visual-auditory cross modal emotional conflict. We selected20college students as subjects, using the word face pictures and two piece of music used in experiment1. It is a2x2x2within-subject design, manipulating the face, the word and the music’s attributes, which are positive and negative. The subjects are told to react at the attribute of the face as positive or negative of the face-word picture with positive or negative music as the background. We record the reaction time, accuracy, and eeg data.20behavioral data and16eeg data are effective. The behavioral result shows there was obvious emotional effect of the visual channel, but the music didn’t educe emotional conflict across visual auditory channel. The eeg results are similar to this. using the270-520ms as time window, that when the word is incongruent with the face, there is a apparent negative wave than it is congruent with the face. That means there is obvious emotional effect within the visual channel, which can be observed as a negative wave in the270-520ms time window, yet music does not educe the visual-auditory cross-modal emotional conflict. It seems that when there are visual channel emotional conflict and visual-auditory cross-modal emotional conflict simultaneously, the subjects’ reaction is submitted to the visual channel’s conflict. However, whether the difference is because of the channels, or it’s the word and music’s own attribute we used in the experiment that matters. As the subjects are much more familiar with the words than the music, and the way they know them are not the same, the intensity of the two kinds of materials’s interference effect are different. This may influence the result of the experiment. Further research should be done to explore more of it.In all, when there are visual channel emotional conflict and visual-auditory cross-modal emotional conflict at the same time, it is too early to say whether the subjects’ reaction is related to the channel effect or the stimulus’ interference intensity. We need further experiments. But by the results’ difference between Experiment1and2. it can be speculated that the channel effect is not enough to bring these differences. Whether the two modal of effects are function independently can not be generalized. either. The reaction of the subjects seems to be more related to the intensity of the interference the two modal brings, which may be related to the interval of the stimulus or the type of materials that used, or both.