Analysis And Application Of Brain Multimodal Medical Image Based On Cycle-consistent Generative Adversarial Networks

Multimodal medical images can reflect the different visual characteristics of different tissues of the human body,and have become the main research objects of artificial intelligence,biomedical engineering,computer science and other disciplines.Lesion segmentation based on multimodal medical imaging can be applied to research fields such as disease diagnosis,preoperative planning,intelligent medical care,and big medical data analysis.In recent years,with the rapid development of deep learning,multimodal lesion segmentation methods based on deep neural networks have made great progress.However,due to factors such as scanning time,acquisition cost,etc.,clinically effective multimodal medical imaging samples are small,which is not conducive to the application of data-driven artificial intelligence algorithms.At the same time,because the lesion is a complex physiological phenomenon,and the image representation of the lesion has the characteristics of specificity,instability,and heterogeneity,the accuracy and universality of lesion segmentation algorithm are low.This thesis addresses the two problems of lack of clinical multimodal medical imaging data and low accuracy of multimodal lesion segmentation.From the perspective of theoretical research and algorithm application,multimodalities of magnetic resonance imaging(MRI)are used as the research objects.Three multimodal medical image generation and lesion segmentation algorithms are proposed based on the cycleconsistent generative adversarial networks(Cycle GAN),and applied to existing open competition data to achieve multimodal image generation and high-precision brain lesion segmentation.The main research works of this thesis are as follows.(1)Perceptual cycle-consistent generative cross-modality segmentation(PerCycle GAN-CMS)is proposed.First,a brain tumor multimodal image generation algorithm based on perceptual cycle-consistent generative adversarial networks(PerCycle GAN)is proposed.Per-Cycle GAN increases the perceptual consistency loss on the basis of the Cycle GAN backbone,builds a new network,and generates high-quality new modal images.The loss of perceptual consistency constrains the mapping relationship between the generated image and the target image at the multi-level semantic level.Secondly,Per-Cycle GAN is combined with the cascaded anisotropic convolutional neural segmentation network to construct Per-Cycle GAN-CMS to segment the lesion of brain tumor.The Per-Cycle GAN-CMS is applied to the dataset Bra TS2015(Multimodal Brain Tumor Segmentation Challenge 2015,Bra TS2015).The experimental results show that(a)the multimodal generation performance of PerCycle GAN is better than that of Cycle GAN.Per-Cycle GAN generates image with three indicators meeting the needs of clinical applications.The peak signal to noise ratio(PSNR),structural similarity index(SSIM)and root mean square error(RMSE)under four conditions are: T1→Flair(23.859,0.867,17.315),Flair→T1(24.546,0.896,16.421),T1→T2(25.544,0.902,14.196),T2→T1(24.206,0.908,17.328).(b)The segmentation effect of the Per-Cycle GAN-CMS algorithm is higher than the that of one single modality.Taking the real T1 + generated Flair VS only real T1 as an example,the index obtained by Per-Cycle GAN-CMS are the Dice coefficient(0.679 VS 0.633),Sensitivity(0.678 VS 0.612),and Hausdorff95 distance(19.487 VS 19.674).It can be seen that the proposed Per-Cycle GAN-CMS can strengthen the detailed information of the lesion,improve the quality of the generated modalities,and improve the objective accuracy of lesion segmentation.(2)A generative multimodal image generation and segmentation network based on dual-scale cross-modality perceptual cycle-consistent generative cross-modality segmentation(Dual CMP-GAN-CMS)is proposed.First,a brain tumor multimodal image generation algorithm based on dual-scale cross-modality perceptual cycleconsistent generative adversarial networks(Dual CMP-GAN)is proposed.Dual CMPGAN uses Per-Cycle GAN as the backbone network,and introduces a dilated residual into the generator to increase the receptive field,resulting in retaining the detailed structure and context information.Secondly,a dual-scale image patch discriminator is constructed,and the generator is optimized by distinguishing patches of different sizes,so that the network can characterize multi-size lesions.Finally,the Dual CMP-GAN is followed by the cascaded anisotropic convolutional neural segmentation network,thus Dual CMP-GAN-CMS is proposed to achieve brain tumor lesion segmentation.The Dual CMP-GAN-CMS algorithm is applied to the dataset Bra TS2018.The experimental results show that(a)in the four cases of T1→Flair,Flair→T1,T1→T2,T2→T1,the objective indicators of the image generated by Dual CMP-GAN are PSNR(23.970,24.564,25.802,24.560),SSIM(0.866,0.894,0.909,0.910),and RMSE(16.979,15.937,13.695,16.112),indicating that the performance of Dual CMP-GAN is better than those algorithms,such as discover generative adversarial networks(Disco GAN),Cycle GAN,et al,.(b)At the same time,the segmentation results of the proposed Dual CMP-GAN-CMS are significantly higher than that of results from one single modality.Taking the real T1 + generated Flair VS only real T1 as an example,the index obtained by Dual CMP-GAN-CMS are the Dice coefficient(0.702 VS 0.619),Sensitivity(0.648 VS 0.529),and Hausdorff95 distance(14.843 VS 18.657).It can be seen that the image generated by Dual CMP-GAN-CMS retains the structure,morphology and connectivity of the brain tumor,and the segmentation performance using the generated modality is close to that of the real one.Therefore,the proposed Dual CMP-GAN-CMS can be used as an alternative brain tumor image generation and segmentation methods.(3)A generative multimodal image generation and segmentation network(Dual CMP-GAN-3D Res U)based on dual-scale cross-modality perceptual cycle consistency and 3D Res U-Net(Dual CMP-GAN-3D Res U)is proposed.First,aiming at the unclear boundary of stroke,a 3D Res U-Net-based stroke multimodal image segmentation algorithm is proposed.3D Res U-Net takes 3D U-Net as the backbone and introduces residuals.The residual structure can more fully extract the features of stroke images and avoid the problem of gradient disappearance caused by the deepening of the network layer.Secondly,the input of the 3D Res U-Net network is changed to randomly offset the sampled image patches centered on the voxel point of the lesion.This kind of image patches can effectively solve the problem of category imbalance.Finally,Dual CMP-GAN and 3D Res U-Net are cascaded to construct Dual CMP-GAN-3D Res U.The Dual CMP-GAN-3D Res U algorithm is applied to the stroke dataset ISLES2015(Ischemic Stroke Lesion Segmentation 2015,ISLES2015).The experimental results show that the multiple segmentation indexes of the proposed Dual CMP-GAN-3D Res U are higher than those of two real modalities.Taking real T1+ real Flair + generated T2 as an example,the Dice coefficient of Dual CMP-GAN-3D Res U is as high as 0.732,which is 3.7% higher than the Dice coefficient(0.706)of real T1+real Flair,which can meet the clinical segmentation requirements for brain images.Therefore,the proposed Dual CMP-GAN-3D Res U algorithm has a high segmentation performance for stroke images with unclear lesion features and unclear boundaries.To sum up,in this thesis three multimodal MRI image generation and brain lesion segmentation algorithms are proposed for solving the lack of multimodal medical images and low accuracy of lesion segmentation.They achieve higher generated image quality and objective segmentation results.The proposed Per-Cycle GAN-CMS can improve the quality of generated modalities while improving the objective accuracy of brain tumor lesion segmentation;the proposed Dual CMP-GAN-CMS can better retain the structure,shape and connectivity of the lesion;and the proposed Dual CMP-GAN-3D Res U can be effectively applied to the segmentation of stroke lesions.Therefore,the three algorithms proposed in this thesis have theoretical innovations and practical application value.