Research On Visual Neural Information Decoding Based On Deep Learning

The unique human brain visual system makes vision the most important way for human beings to obtain perceptual information facing the external environment.Deciphering the working mode of the human brain with the help of computers and studying the visual processing mechanism of the human brain has always been a hot topic.Predicting the corresponding external visual stimuli using neural signal patterns plays an important role in understanding the interaction of real visual scenes.At present,there are still some challenges in the research of visual neural information decoding based on Electroencephalography(EEG),such as the experimental paradigm for collecting EEG signals is not rigorous enough,the effective features of EEG signals cannot be fully utilized,and the accuracy of classification needs to be further improved.This thesis designs an experimental paradigm that meets the requirements of neuroscience,collects EEG signals,combines computer vision with human vision to propose a neural network model suitable for classifying brain activity patterns induced by visual stimuli to study the decoding of visual information,and then explores the visual processing mechanism of human brain.The research content can be divided into the following three aspects:1.An experimental paradigm is designed to collect EEG signals induced by image visual stimuli,which meet the requirements of neuroscience and minimize the interaction between visual stimuli to the greatest extent.EEG signals of the subjects under the current experiment are collected based on EEG technology.In order to obtain relatively pure EEG data for subsequent use and analysis,the collected EEG signals are preprocessed by transforming reference electrodes,filtering,removing eye-movement artifacts,condition segmentation,etc.,and then the dataset is constructed.2.In order to prove the effectiveness of the experimental paradigm,two different schemes are adopted to construct the datasets for the black screen segments in the experimental paradigm,and a hybrid neural network model based on Convolutional Neural Networks(CNN)and Long Short-Term Memory(LSTM)is proposed to classify the EEG data under different schemes.The experimental results show that the setting of black screen segments plays a positive role in avoiding the time correlation between EEG data.3.A novel spatio-temporal convolutional neural network framework with nonlocal attention mechanism is proposed for image-induced brain activity pattern classification,which considers the temporal,spatial and global features of EEG signals.The framework extracts temporal features of EEG signals at multiple scales,and correlation features of all EEG channels from a spatial perspective.More importantly,the framework utilizes non-local feature extraction operations to realize the global information interaction between EEG signals over long distances and enhance the feature expression of some important information.The experimental results show that the proposed model performs better than other advanced deep learning algorithms on MNIST-EEG and Image Net-EEG datasets,and the classification results are 24.00% and50.10%,respectively.It also analyzes which brain regions are activated over time.The results show that the occipital region plays a crucial role in processing visual tasks,and the temporal region and some frontal regions also play an important role in the later stage of visual decoding.