Research On The Key Problem Of Audiovisual BCI Speller With Covertattention

Character input system using Electroencephalogram( EEG) provides new communication to users with severe motor disabilities through virtual keyboard. The performance of early study,however, depends critically on the user’s ability of gaze movement and could not accommodate users with impaired ocular motor control. To solve this problem, researchers gradually search alternative BCI(Brain-Computer Interface)paradigms with covert attention, among which multimodal paradigms with visual and auditory stimuli were given high expectation and related research needs urgent in-depth study.Based on the above background, in this dissertation three kind of gaze-independent paradigms were firstly designed, which are visual stimuli paradigm, auditory stimuli paradigm, and audio-visual combined paradigm. The role of visual and auditory stimuli and the brain response to each paradigm were investigated through the analysis of different paradigms with temporal-spatial distribution of Event-Related Potential(ERP) components and distribution of the source location. From the comparison of the spelling speed and workload, it was proved that audio-visual combined paradigm was a low workload and effective character input system.The effect of temporal congruity on audio-visual combined gaze-independent character input system was designed and studied through five different visual-to-auditory delay paradigms. By analyzing the ERP components in different conditions, we found that early ERP components of audio-visual combined paradigm were related to both visual stimuli and auditory stimuli, whereas the latency of the ERP components was mainly affected by the onset of visual stimuli. However, conditions with limited visual-to-auditory congruity(less than 100 ms) could all have higher classification and no significance was found among conditions. It proved the robustness of the classification algorithm used in this study and indicated that the combined system could achieve higher accuracy within certain range of visual-to-auditory delay and proper classifier. So there is no necessary for too precise demand of visual-to-auditory congruity.At the end, a novel audio-visual associated paradigm with covert attention was designed, which is audio-visual parallel paradigm. This paradigm made it come true that choosing the target character through independent and parallel visual and auditory task. The feasibility of this paradigm was also proved through the analysis of the ERP characteristic and classification result. Results showed that this parallel paradigm could produce stable ERP with high discrimination. More than 70% subject could get online character input accuracy of more than 80%.In short,different audio-visual paradigms with covert attention were designed in this dissertation, the role of visual and auditory stimuli were studied deeply, and the advantage of audio-visual combined paradigm as well as the feasibility of audio-visual parallel paradigm was proved. This research achievement could hopefully get further popularization and application in BCI system for severe paralyzed patients.