Research On Affective Computing In Videos Based On EEG And Brain Encoding Decoding

As a discipline spinning computer science,psychology and cognitive science,affective computing is related with human affective activities including emotion arousing,transferring and maintaining.Recently,emotion identification/tagging,being considered as a theme of interested in affective computing,is regarded as an important research field.Emotion identification/tagging using physiological signals,or implicit emotion tagging,attracts researcher's attention and publications increases in the past years.However,there are some issue need to be addressed.Firstly,multi-channeled electroencephalography(EEG)enables researchers to observe cortical activities in whole brain scale but currently the modelling methods only employ simple combination of features from single channel.Secondly,implicit emotion tagging requires physiological signals of subjects,which are acquired in price of high cost and long time consuming.Thirdly,several types of EEG features have been proposed in the published works.However,the comparisons among these types of features are insufficient thus few guiding for better extracting information from EEG singles are available for upcoming research works.The following researches are conducted to overcome these issues: 1)A functional network brain model is proposed;2)a novel framework for emotion tagging is proposed to reduce the dependence on physiological signals of subjects' responses;3)a framework of feature benchmark is proposed.The contributions in this thesis are listed as follows:1.Proposed a functional network brain model,which is for describing emotional activities,based on connectivity between electrodes in EEG.For the inability of current works using signal channel features to describe relationship between cortical areas,a model based on connectivity are proposed.To describe cortical areas corresponding to electrodes,three types of connective features,namely,Pearson correlation coefficients,phase coherence and mutual information are used.The explanations of the three kinds of connectivity are also given.For the influence of different bands of the signals,connective features extracted corresponding to different bands and connective type are used.The experimental results indicates that comparing with existing work,the model based on connective features give better results.2.A brain encoding/decoding based multimedia emotion identification/tagging method is proposed.For the limitation caused by high time and economic consumption of acquiring physiological signals of subjects' responses,a new framework of emotion identification/tagging is proposed.In the proposed framework,brain encoding/decoding conception is introduced and the missing subjects' responses for the corresponding multimedia contents are completed via the conception.Then based on the completion the emotion identification/tagging tasks are conducted.The proposed framework is able to reduce the costs and time of physiological signal acquiring.It is also able to extend the application of existing affective database.Experimental results indicates that the performance of the proposed framework outperforms the performance of multimedia contents.Additionally,analysis are conducted on the factors involved in the proposed framework.Stimulus features and policies of target cortical features selection are two factors of the proposed framework in extending application of affective computing database.For stimulus feature generation,an analysis is conducted on the influence of existence of audio signal features on the performance of the framework.For brain encoding model training,the influence of different target cortical selecting policies on the performance of the framework is analyzed.The two analysis are based on the experimental results of the proposed emotion tagging framework.3.A framework of feature benchmark is proposed for comparing the existing EEG features.The proposed framework employ linear machine methods to eliminate influence introduced by factors except EEG features.Verifying tasks are conducted to validate the validity of the proposed framework.The experiment results indicate that the results of comparison under the proposed benchmark framework are similar to the results of verifying task.This means that the proposed benchmark framework is able to provide guiding information for EEG signal processing part in affective computing tasks.