Research On Internet Meme Classification Based On Deep Learning

Internet meme,commonly known as "biaoqingbao" or "gengtu".Communities and platforms that are popular in the Chinese Internet,especially among young people,are widely used and have great influence.In fact,this situation is also a common phenomenon in the Internet culture of various linguistic and cultural backgrounds.On the one hand,Internet memes contain a lot of real and accurate public opinion information to be mined,and the classification of sentimental tendencies can be widely used in the investigation,prediction and feedback of various information.On the other hand,Internet memes are also increasingly used to disseminate harmful content in a more implicit and obscure way,such as personal attacks,dissemination of discrimination and hatred,which need to be reviewed and filtered in an automated way.In addition,since Internet memes are essentially composed of information in two modality of image and text,it has to be said that Internet memes provide an excellent research object for multi-modality learning.Since 2020,various classification tasks of Internet memes have received more and more attention.Although Internet memes in languages such as German,Thai and Greek have also been studied,the main research object is still the Internet memes in English.In particular,Chinese is one of the most populous languages in the world,but the classification of Chinese Internet memes is still extremely lacking.In this thesis,three tasks about the classification of Chinese Internet meme pictures are set.First,the classification task of Internet meme and non-meme Internet pictures in the Chinese Internet community environment;The second is the classification task of positive,neutral and negative sentimental tendencies of Chinese Internet memes;Third,the Chinese Internet memes are classified into 7 harmful categories from harmless to harmful.Taking the above Chinese Internet meme image classification task as the main research content,the main work and contributions of this thesis are as follows:(1)A Chinese Internet meme multi-modal data set that can be used for the above classification tasks is constructed.Before that,there was no relevant data set available,so this thesis built a data set with a total of30000 pictures through Internet crawling,similarity de-duplication,manual filtering,label labeling and text information extraction for the three classification tasks studied in this thesis.(2)A self-learning decision level multi-modal fusion model is proposed.In this thesis,the Res Net50 and BERT models,which are more suitable for single-modality classification of images and texts in Chinese Internet memes,are compared.Then on the basis of these two singlemodality models,a self-learning decision level multi-modal fusion model is designed.This model can effectively combine the output results of each single-modality to make the final multi-modal decision output,and the experimental results of each single-modality model are improved.(3)A feature-level multi-modal fusion model of cross-modal attention mechanism is proposed.The model uses Res Net50 and BERT to extract the features of image and text modalities in Internet memes respectively.Then the attention mechanism is used to fully extract the key information of multi-modal joint representation features and cross-modal related information at the same time of modal feature fusion.Compared with the feature-level fusion model without attention mechanism,this model has achieved better results in various classification tasks and is slightly better than the decision-level fusion model in overall performance.(4)In view of the non-discrete classification tasks of Internet meme sentimental tendencies classification and harmful degree classification,this thesis proposes a more suitable evaluation index NDC score for this type of classification task.This indicator is more objective and effective than the conventional F1 score in evaluating the performance of a model on non-discrete classification tasks.(5)This thesis makes an exploration,and tries to apply the selflearning decision-level fusion model that has achieved good results on Chinese Internet memes and the feature-level fusion model of cross-modal attention mechanism to the classification experiment of sentimental tendencies on an open English Internet meme data set.The experimental results exceed the effect of the baseline model of the data set,and preliminarily verify the versatility of this model in cross-language.