Multi-view Deep Learning By Mining Views' Complementary Information Via Nonlinear Mapping

Multi-view learning is derived from multi-view data,and attempts to generate a model with a better performance by exploiting information among multi-view data.Until now,nearly all relevant literatures of multi-view learning assume that consistency and complementarity,which separately represents the common and specific information of multiple views,are two main underlying properties of the multi-view data.While many advanced multi-view learning algorithms only focus on consistency or complementarity to improve the performance of learning tasks.Thus,the multi-view deep learning,which learns the consistency and complementarity of multi-view data simultaneously,is still an open question in the multi-view learning.In order to construct a deep network learning the consistency and complementarity of multi-view data simultaneously,this paper proposes a benchmark network,which learns the consistency by the encoder-block,mining complementarity by max-pooling,mean-pooling,or weighted-summation,generates feature vectors containing consistency and complementarity,and finally makes prediction through the full-connected layer.The benchmark model can only explore a kind of complementary.To explore multiple kinds of complementary,this paper also proposes a multi-view network with self-attention mechanism,which is based on the benchmark model,and makes it to extract multiple feature vectors containing different complementary information,finally concatenates all feature vectors to produce an augmented vector.The benchmark model and the multi-view network with self-attention mechanism both mine the complementarity information through vector operations,rather than matrix operations.In order to make use of matrix operations,this paper combines the benchmark model and the capsule network to propose the multi-view capsule network.Multi-view capsule network mines complementarity through the dynamic routing process between capsule layers and makes classification by prediction matrix.To compare the performance of each model,all models conduct experiments on the same data sets,and extensive experiments conducted on eight real-world datasets have demonstrated the effectiveness of our proposed methods.Finally,the multi-view capsule network has achieved the best prediction results on all experimental data.