Image Multi-label Classification In Medical Literature Based On Transfer Learning

Making full use of image resources and retrieve them efficiently becomes focus and hotspot because medical workers and researchers have the larger and higher demand of medical multimedia resources.Automatic annotation of medical image's modality is the foundation of medical image retrieval.Image resources in literature is becoming more and more important due to they are rigorous,scientific and professional.However,existing researches mainly mined web resources and small number of labeled data from hospital,and most of the past researches only used visual information representing the underlying semantics.Thus "Semantic Gap" becomes obstacle between between user's demand and retrieval results.Generally,annotation of medical images is multi-labels task,so there is a problem about data imbalance.In order to promote professionalism in mining medical image resources,this paper annotates image resources in literature.And utilizes complementarities of textual and visual information to improve high level semantic representation in image annotation.As to the problem of the small amount of medical labeles data,this paper adopts transfer learning to avoid over-fitting in deep learning.What's more,this paper use upsampling and transfer learning to solve data imbalance.Firstly,use alt text-based textual upsampling and GAN-based image upsampling to balance data.Secondly,respectively adopt emerging technologies,Res Net and BERT model to transfer knowledge in image data and text data.Then label data through fusing two models' outputs.Last but not least,the method propoesd from this paper has reached hamming loss of 0.0143,macro-average F1 score of 0.4932,micro-average AUC of 0.9501 and macro-average AUC of 0.9024.