Nuclei Segmentation Based On Semantic Image Synthesis And Masked Image Inpainting

Pathological image is the gold standard of cancer diagnosis,and the morphological and texture features of nuclei in pathological image are an important basis for cancer diagnosis.Extracting these features needs to segment the nuclei first.Therefore,how to effectively improve the performance of nucleus segmentation has become an urgent research topic in the field of cancer diagnosis.With the development of pathological tissue sectioning technology and digital imaging equipment,the number of pathological images increases exponentially.On the contrary,there are limited nuclear labeling data.The whole section pathological image has high resolution and complex image content.It is urgent to segment the nuclei from the pathological image and carry out a series of quantitative analysis by automatic computer-aided technology.However,if we want to use advanced nuclei segmentation technology,a large number of finely annotated segmentation data is essential.Because of such data contradiction,this paper proposes the work of pathological image generation.By generating the corresponding pathological image from the semantic segmentation tag,we can obtain a high-quality pathological image nuclei segmentation data set without paying any labeling cost.On the other hand,the dilemma of data scarcity forces researchers to strengthen the ability of data utilization.This paper proposes an auxiliary task of mask image inpainting to help the segmentation network learn high-resolution nuclear features,so as to effectively improve the performance of nuclear segmentation on the premise of few samples.Based on the above two research ideas,the achievements of the paper are as follows:Firstly,a semantic image synthesis network SC-SEG-GAN for pathological image nuclei segmentation is proposed.In view of the problems of data scarcity,rough annotation and large difference in the number of nuclei of various cancers in the existing pathological image nuclei segmentation data sets,this paper proposes a generation network SC-SEG-GAN for generating corresponding pathological images from semantic segmentation labels to effectively promote nuclei segmentation.The network uses spatial condition convolution,spatial condition normalization,discriminator based on segmentation network and the auxiliary cancer classification module,the pathological images of each cancer species are generated,which are realistic and highly aligned with the semantics of segmentation labels.Experiments on the nuclei segmentation data sets of multi-organ pathological images of Mo Nu Seg and Cryo Nu Seg show that adding the data generated by SC-SEG-GAN has generally improved the performance of major segmentation networks,especially on cancer with few samples.Compared with traditional data augmentation methods on the Mo Nu Seg data set,SC-SEG-GAN can not only improve the performance,but also achieve higher segmentation performance by combining with traditional methods.Secondly,a nuclei segmentation framework MII-SEG combined with mask image inpainting task is proposed.The framework draws lessons from the idea of masked autoencoders MAE,a self supervised image depth feature learning method,by randomly covering some nuclei in pathological images,the network can learn the characteristics of nuclei during image reconstruction.Through joint training with segmentation network,the nuclear features in the features extracted by segmentation network are enhanced,so as to effectively promote the task of nuclei segmentation.Experimental results on two datasets,Mo Nu Seg and Cryo Nu Seg,adapted with three different architectures of segmentation networks,Unet,UNe Xt and UCTrans Net,show that the performance of the networks adapted according to the ideas of the segmentation framework MII-SEG is generally improved compared to the pre-adaptation,verifying the broad validity and efficient performance improvement capability of this segmentation paradigm.