Visual Encoding And Decoding Method For Human Brain Activity Evoked By Natural Images

In recent years,brain science and brain-like research have become hot topics in the field of science and technology and health.In this context,analysis,simulation and interpretation of the visual information processing mechanism of human brain based on mathematical analysis and computer simulation is important for the development of brain science,artificial intelligence and brain-computer interface.In related studies,researchers have built encoding models to simulate the activity patterns of the visual cortex,or built decoding models to know what people see from human brain signals.However,previous studies have often focused on encoding or decoding tasks alone,and the encoding and decoding have not been well integrated.In this thesis,we will focus on the information processing mechanism of visual cortex,and explore encoding and decoding in a parallel way,that is,we understand the information processing mechanism of visual cortex and interpret visual signals at the same time.In this way,we can verify the encoding theory with the aid of decoding and enhance the interpretability of decoding method with the support of encoding theory.Our works and contributions are as follows:We proposed a sparse encoding model based on Gabor-like features according to the encoding characteristics of early visual areas.The model extracts the sensitive features of early visual areas from images through a set of feature transformations,and then builds a mapping between the features and brain activity through sparse optimization.In this setting,we can inverse the mapping between the features and brain activity by solving a quadratic optimization,and achieve identification of visual stimuli(that is,to find the visual stimulus that the subject watched from a given set according to fMRI signals).Experiments show that the proposed feature transformations can significantly improve the encoding accuracy,and the identification accuracy is more than 92%.We further improved the encoding procedure to make the whole encoding procedure reversible,and then designed the corresponding inverse procedure to achieve the reconstruction of visual stimuli(that is,to reconstruct the visual stimuli that the subjects watched according to fMRI signals).The reconstruction method has strong interpretability.Experimental results show that the reconstruction results retain the basic outlines of visual stimuli.Based on the similarity of feature representation between deep convolutional neural network(DCNN)and human ventral pathway,we build encoding models for early visual areas,V4 and high-level visual areas to achieve visual encoding.In addition,we use neural network visualization technology to reverse the encoding process and achieve the reconstruction of visual stimuli.The experiments show that the shared features extracted from DCNN can successfully encode the voxels in early visual areas,V4 and high-level visual areas,and the reconstructions retain the structural and semantic information of the original stimuli.Based on the related researches of cognitive neuroscience,we proposed a brainlike encoding model based on DCNN framework.The model can simulate the information processing mechanism of visual pathway in early visual areas,meanwhile,it can be trained on a given fMRI data set.The encoding process can be reversed,and the stimuli can be reconstructed by using a deep generation network and back propagation.The experiments show that the encoding model reaches high predictive accuracy,and the feature representation conforms to the relevant researches of visual cortex,and the reconstruction results also retain the structural information of visual stimuli.