A Study On Audiovisual Brain-computer Interface(BCI) And Its Clinical Applications

Brain-computer interfaces(BCIs)provide non-muscular communication and control by directly translating brain activities recorded from the scalp into computer control sig-nals and thus enable users with motor disabilities to convey their intent to the external world.An important issue in the BCI research is improving the performance of BCIs,which enables considerably enhanced interfacing between the user and the machine.Po-tential applications include BCI controlled wheelchair and neuroprothesis,and conscious-ness detection in patients with disorders of consciousness(DOC)etc.The main contribu-tion of this study is making a research on high-performance BCIs for DOC patients,and applications in detecting awareness and assisting clinical assessments for patients with DOC.First,we proposed a novel audiovisual BCI system in which spatially,temporally and semantically congruent audiovisual stimuli of numbers were employed.The subjects were instructed to focus on the target stimuli cued by an instruction and to ignore the nontarget stimuli.Ten healthy subjects were included in the experiment to evaluate the proposed system.The results indicated that the audiovisual BCI system outperformed the auditory-only and visual-only systems.Furthermore,we observed audiovisual inte-gration effects for the attended target stimuli but not for the nontarget stimuli.These effects enhanced the discriminability between the brain responses for target and nontar-get stimuli.Next,the proposed system was applied to detect awareness for seven DOC patients,and five of the patients showed command following.We first applied an audio-visual BCI for awareness detection in patients with DOC and the results indicated the BCI system could be used as a supportive bedside tool for awareness detection in patients with DOC.Second,we designed an audiovisual BCI system using temporally,spatially,and se-mantically congruent audiovisual stimuli(i.e.,visual and spoken "yes" or "o").This BCI was used to supplement the JFK CRS-R in assessing the communication ability of patients with DOC.The patients were instructed to select the answers("yes" or "no")using the BCI,based on the situation-orientation questions.Thirteen patients with DOC participated in our experiments involving both BCI-and JFK CRS-R-based assessments.Among these patients,one MCS patient obtained responsive results in both the JFK CRS-R-and the BCI-based assessments,and five patients(four VS and one MCS)ob?tained unresponsive results in both assessments.Seven patients(four VS and three MCS)obtained unresponsive results in the JFK CRS-R-based assessment but responsive results in the BCI-based assessment.Furthermore,four of the seven patients later got improved scores in the JFK CRS-R-based assessment during their rehabilitation procedure.These results showed that our BCI-based communication assessment method was more sensitive than the JFK CRS-R.With this study,we won the second place of the video showing event in the 2rd China BCI Competition(2015).Next,a novel 3D BCI based on stereo stimuli was proposed to supplement the JFK CRS-R in assessing the object recognition of patients with DOC.The 3D visual scene is more realistic and attractive than the 2D visual scene,thus we used stereo stimuli in the audiovisual BCI system.Four healthy subjects participated in the experiment to evaluate the system in the 2D and 3D modes.The results indicated that the proposed 3D BCI based on stereo stimuli outperformed the traditional 2D BCI.Then,this system was applied to assess the object recognition of patients with DOC.Seven DOC patients’results in the two assessments indicated the combination of the JFK CRS-R and BCI assessments may provide more precise and objective diagnosis results compared with the JFK CRS-R alone.Finally,we designed an audiovisual BCI-based communication system for patients with DOC.The system integrates kinds of questions through a multi-level graphical user interface(GUI),and uses a variety of feedback to attract the patients’ attention and to make the communication more interesting.During communication,a question was first presented to the patients by the proposed audiovisual BCI,then the patients were instructed to focus on the button corresponding to the answers("yes" or "no")and count its flashes.Lastly,the result was achieved by our online detection algorithm using the ongoing EEG.After each trial,we added the features and labels of this trial into the training dataset,and updated the classification model to improve the online accuracies.Two out of five patients with DOC experienced the communication system could use this communication BCI system to exchange messages with the outside world.