Research On Multimodal Recognition Model Based On Deep Associative Memory Network And Multidimensional Weights

The information in the real world often appears in many different forms at the same time.Researches on Cognitive science show that in the process of knowing the world,creatures combine the information from multiple senses to make a comprehensive judgment,so that they can obtain a general understanding.The information from different senses are complementary to each other.Associative memory is an important function of the creatures' brain,to learning a new thing,creatures will get many information from different senses and form memory to storage multiple modal information.After that,whenever they encounter some information about the thing they have memorized,the other relevant information will be recalled.With the deepening of the study of nervous system,people realize that the brain's learning of new things is a process of constantly extracting abstract concepts,abstracting information from the original state to a higher level,and then connecting many types of the high-level abstraction together to form associative memory.In order to simulate the ability of creatures that make use of multi modal information to study and form associative memory,An deep associative memory network based on Deep Belief Network and Bi-directional Associative Memory neural network is presented,and a multi-modal recognition of multidimensional weights based on the such network.The deep associative memory network can extract the features of each modal through the deep belief network,and then use the bidirectional associative memory network to establish the connection between multiple features.The establishment of this connection and the two-way generation of the deep belief network enable the model to reconstruct the missing information.The multi-modal recognition of multi-dimensional weights based on the depth associative memory network is not only has the higher accuracy of classification than the models before,but also capable of performing well on classification tasks even when some signals are missing.In this paper,we provide image signal and audio signal in the experiment,and the network learns based on them.The experimental results verify the ability of the deep associative memory network to regenerate the missing signal.By comparing with many traditional models,we validate the validity of our multi-modal recognition model of multidimensional weights,and due to the multidimensional weights other than singular weights,our model have strong stability when identifying different categories.