Bimodal Emotion Recognition System Based On EEG And Facial Expression

With the development of artificial intelligence technology,quickly and accurately identifying emotions can help robots perceive the state of users,and emotion recognition has become an increasingly important research topic.Recognizing emotions from only one modal has its shortcomings.Starting from multimodal,effective information of different modals can be extracted to improve the accuracy of emotion recognition.This paper combines two modals of Electrocephalogram(EEG)and facial expressions,and implements a bimodal emotion recognition system based on the Android platform,which can realize the two-category emotion recognition of happiness / sadness,and can achieve preprocessing and feature extraction on the Android system With the functions of fusion and SVM classification,the graphical interface is used to display the collected EEG waveforms and images of the face area,and the sample data recorded by the system is displayed.The main work of this paper is as follows:(1)Aiming at the EEG modal,Wavelet Threshold Denoising(WTD)is used for preprocessing,and then Fractal Dimension(FD)and Multiscale Entropy(MSE)algorithms are used to extract EEG signal features.In order to verify the rationality of the EEG signal preprocessing and feature extraction methods,this paper conducts emotion classification on the EEG signal in the EEG emotion database——DEAP database,With only one EEG channel FP1,the accuracy of Support Vector Machine(SVM)classification can reach 74.9%.(2)Aiming at the facial expression modal,Histogram Equalization(HE)is used for preprocessing,and the uniform pattern Local Binary Pattern(LBP)algorithm is used to extract facial expression features.In order to verify the rationality of the facial expression preprocessing and feature extraction methods,this paper also performs emotion classification on the Japanese Female Facial Expression Database(JAFFE),It also proves that using uniform pattern LBP feature can recognize emotion more accurately.(3)A bimodal emotion recognition system based on the Android platform is designed in this paper.The system uses a Bluetooth EEG acquisition headgear to collect EEG signals and transmit them to the Android system analysis and processing platform.A visible database and a training set that can be deleted are designed.User can train the SVM model and set the SVM parameters,which is convenient and user-friendly.(4)This paper designs an experiment to test the performance of emotion recognition of the bimodal emotion recognition system based on the Android platform,and conducts a comparative experiment of bimodal,EEG modal,and facial expression modal.The experimental results show that the recognition accuracy rate of positive and negative emotions of this system reaches 73.75%,and the system is feasible.At the same time,the comparison experiment proves that,compared with the traditional method of recognizeing emotions through EEG signals or facial expressions,the form of bimodal can extract more emotion information,and the effect of emotion recognition is better.