The Cognitive And Neural Mechanism Of Audiovisual Temporal Order Perception

Time is always a mystery bewildering philosophers and psychologists for long periods. It is important in human behavior, cognition and consciousness. The concept of time is complicated and multiplex. Here the focus of this dissertation is on relative timing or the perception of temporal order instead of interval timing or the perception of duration. Temporal order perception or relative timing refers to the ordinal relationship between events, i.e., the sense of synchrony, successiveness and order when two events occur concurrently or successively. The ability to perceive order lies fundamental in speech comprehending, music appreciation, sports exercises, and etc. Generally, the four studies included here using behavioral and electrophysiological methods addressed different theoretical issues about temporal order perception in audiovisual domain.First, perceptual synchrony and multisensory integration both vary as a function of stimulus onset asynchrony, but evidences from behavioral, patient, and lesion studies all support some dissociation between these two processes. Although it has been found that both perceptual synchrony and multisensory integration are recalibrated after exposure to asynchronous multisensory stimuli, no studies have directly compared these two recalibration patterns. We addressed this in Study lby using McGurk speech and requiring participants to perform simultaneity judgments and a syllable identification task in separate sessions. The results revealed that after exposure to asynchrony, both perceptual synchrony and McGurk fusion shifted towards the temporal lag. The recalibration aftereffects (i.e., the magnitude of shifts) of these two processes have no significant difference and correlation. In addition, McGurk fusion increased strongly at the direction of the temporal lag, which could not be fully explained by fusion shifts. Thus, the present research implies that recalibration patterns of explicit and implicit timing represented by perceptual synchrony and multisensory integration have both similarity and difference.Second, out-of-synchrony experiences can easily recalibrate one’s subjective simultaneity point in the direction of the experienced asynchrony. Although temporal adjustment of multiple audiovisual stimuli has been recently demonstrated to be spatially specific, perceptual grouping processes that organize separate audiovisual stimuli into distinctive "objects" may play a more important role in forming the basis for subsequent multiple temporal recalibrations. We investigated in Study 2 whether apparent physical differences between audiovisual pairs that make them distinct from each other can independently drive multiple concurrent temporal recalibrations regardless of spatial overlap. Experiment 1 verified that reducing the physical difference between two audiovisual pairs diminishes the multiple temporal recalibrations by exposing observers to two utterances with opposing temporal relationships spoken by one single speaker rather than two distinct speakers at the same location. Experiment 2 found that increasing the physical difference between two stimuli pairs can promote multiple temporal recalibrations by complicating their non-temporal dimensions (e.g., disks composed by two rather than one attributes and tones generated by multiplying two frequencies); however, these recalibration aftereffects were subtle. Experiment 3 further revealed that making the two audiovisual pairs differ in temporal structures (one transient and one gradual)was sufficient to drive concurrent temporal recalibration. These results confirm that the more audiovisual pairs physically differ, especially in temporal profile, multiple temporal perception adjustments will be more likely content-constrained regardless of spatial overlap. These results indicate that multiple temporal recalibrations are based secondarily on the outcome of perceptual grouping processes.In the third study we investigated whether temporal binding by space is superior to by feature in a scenario where a transient sound synchronizes with only one of two successive flashes. In Experiment 4, the successive flashes with different colors were either overlapped or separated and participants had to indicate the color of the flash which is synchronous with the sound regardless of their positions, while in Experiment 5, the two flashes were always separated laterally but with same or different colors and participants had to indicated the position of the flash synchronous with the sound regardless of their colors. Audiovisual temporal binding by color were severely impaired when the flashes came from identical rather than different positions whereas temporal binding by position were much facilitated when the flashes were of the same rather than different colors. A between-experiment comparison revealed more accurate performance when temporal binding was made only by positions than colors. The results showed that audiovisual temporal binding by space is superior to by feature, implying that time and space are more tightly interconnected and temporal binding by space may occur at an earlier processing level.At last, it has been acknowledged that the activity of pre-stimulus oscillation in the brain is fluctuated but correlated with following behavioral and perceptual performances. Here using power spectrum analysis of electroencephalogram the last study explored how pre-stimulus oscillation activities in humans modulate their subsequent discrimination of perceptual simultaneity from non-simultaneity in audiovisual domain. Oscillations within the beta and gamma range (15～19 Hz and 42～ 45 Hz) were significantly weaker before physically auditory lagging stimuli which was judged as simultaneous relative to as non-simultaneous, whilst oscillations within the range (23～28 Hz and 45～48 Hz) were significantly stronger before physically auditory leading stimuli when it was judged simultaneous rather than non-simultaneous. Furthermore, the pre-stimulus beta and gamma powers separately predicted the subjects’ reports of simultaneity for these physically different stimulus sequences despite with opposite linear trend. Our results indicate that the fluctuations of ongoing oscillation are able to shape the following perceptual simultaneity in audiovisual domain and that the subjective simultaneity is actually not homogenous given that the two physical sequences both likely to be perceived as simultaneous correspond to different neural substrates.As a summary, the first study revealed that recalibration patterns of explicit and implicit timing represented by perceptual synchrony and multisensory integration have both similarity and difference. The second study found that apparent physical differences between audiovisual events especially differences in temporal profile can drive multiple temporal perception adjustments regardless of spatial overlap. The third study demontrated that spatial clues faciliate the temporal binding by feature clues while feature clues interfere the temporal binding by spatial clues. The fourth study found that prestimulus high band oscillaton can successfully preidct the following simultaneity judgment but the predictors for physically auditory leading and lagging stimuli are separate. All of them provide new evidence for distributed timing and some implications for the basical timing mechanism associated with spatial dimension and the effect of high cognitive factors on timing (e.g., perceptual grouping processing) which determine the finally reported temporal perception.However, there are more unsolved problems than solved which await future researches, such as the neural basis for temporal binding by spatial and feature clues, and the role of crossmodal perceptual grouping on intersensory timing in cluttered multisensory scenarios. And it will be of merit to extend the theoretical studies on time into applications, for instance, into the virtual reality, the observer-moving, and the social interactive scenarios.