The Effect Of Attention On Multisensory Integration

Multisensory integration generally refers to the set of processes by which information arriving from the individual sensory modalities (e.g. vision, audition, touch) interacts and influences processing in other sensory modalities, including how these sensory inputs are combined together to yield a unified perceptual experience of multisensory events. By combining information from different modalities, multisensory integration may help us induce the noise of perception system, and help us perceive information better. It often showed that judgments of multimodal information presented simultaneously are faster and more accurately on the behavior.Reviewing studies on multisensory integration, we could find that the early studies paid more attention to the characteristics and processing methods of multisensory integration itself. In recent years, researchers have increasingly turned to the point of interest on the relationship between attention and multisensory integration. However, studies have mostly discussed whether the condition of attention and non-attention could have influence on multisensory integration ignoring the flexibility that attention could point to the specific modality. The present study aimed to examine whether and how the attention pointing to the specific modality effected multisensory integration.Three studies including a total of six experiments were carried out.In study 1, by directing attention to different modalities (visual, auditory, audiovisual) by cues, the present study investigated the effect of attention pointing to the specific modality on multisensory integration. Two experiments were included. In experiment 1, simple graphics (squares and circles) and tones were used as materials, and participants’task was to make the appropriate response to target stimuli according to cues. It showed a processing advantage for the bimodal targets only when the attention was divided between two modalities, the so-called redundant signal effect. To account for the redundant signal effect, the distributions for bimodal responses were compared to the race model. We found that the redundant signal effect of bimodal targets in the divided attention condition arose from the multisensory integration. In one word, the multisensory integration occurred when the attention was divided between visual and auditory modality. Experiment 2 used squares and circles as visual stimuli and spoken words as auditory stimuli to investigate the effect of attention on multisensory speech integration. The visual and auditory stimuli were semantically congruent or incongruent. The results from a discrimination task showed that in the condition of divided attention, participants required less time to react to audiovisual semantic-congruent targets than other types of targets--that was the redundant signal effect. According to the race model, the redundant signal effect in the condition of divided attention originated from the multisensory integration.On the basis of study 1, study 2 examined the influence of attentional load on multisensory integration. Two experiments were included. Experiment 3 found that only in the absence of attentional load condition not in the no-load condition, it produced the redundant signal effect. We found that the redundant signal effect of bimodal targets in the divided attention condition arose from the multisensory integration. Experiment 3 showed that even in the condition of divided attention, multisensory integration would be affected by visual attentional load. Experiment 4 found that whether or not presented auditory load, participants made fastest responses to the audiovisual targets, and that was the redundant signal effect. We found that the redundant signal effect of bimodal targets in the divided attention condition arose from the multisensory integration. Experiment 4 showed that in the condition of divided attention, auditory load had no influence of multisensory integration.Using audiovisual cues with rhythm, study 3 investigated whether and how the fluctuation of attention affected multisensory integration and its neural mechanisms. Experiment 5 set the circumstance in which targets was match with rhythms (tune), not match (out of step) and non-rhythm (silence). It found that only in the tune condition participants made fastest responses to the audiovisual targets. We found that the redundant signal effect of bimodal targets in the tune condition arose from the multisensory integration. On the basis of experiment 5, using event-related potentials technique which has a high time resolution, experiment 6 examined the effects of fluctuation of attention on multisensory integration and its neural mechanisms. It found that, for the auditory targets, N1 amplitude in the tune conditions was significantly greater than in the silent condition; and at Pz and P3 electrode point, P2 amplitude in the tune conditions was significantly greater than in the silent condition. For the visual targets, N1 amplitude in the front and occipital scalp in the tune conditions was significantly greater than in the silent condition; P2 amplitude in the frontal region and right middle frontal in the tune conditions was significantly greater than in the silent condition. Based on the above results, there were N1 attentional effects regardless of auditory or visual targets. This showed that participants intended to pay attention to targets in the tune condition than in the silence condition. According to the data processing method in the existing studies of multisensory integration, we analyzed data for each 20ms of ERP amplitude within the time window of 0-500ms. We compared the sum of ERPs of visual and auditory targets (A+V) and ERPs of audiovisual targets (AV). The results showed that only in the tune condition, in the right brain within 121-140ms, and in the central brain within 141-160ms, AV is greater than A+V. That was called super-addictive effect. This suggested that in the present condition, multisensory integration would be produced when the targets were on the peak of fluctuation of attention, but this effect was not sustained. It would occur within the time window of 121-160ms after targets were presented.In summary, attention had influence on multisensory integration, which was reflected in multisensory integration would occur only when in the divided attention condition. Visual and auditory attentional load had asymmetry effects on multisensory integration, which was reflected only visual attentional load had influence on multisensory integration but not auditory attentional load. The fluctuation of attention had effects on multisensory integration. It showed that audiovisual information would be integrated only when targets were presented on the peak of attention and this effect was not sustained. It would occur within the time window of 121-160m.