Study And Application Of Deep Learning Based Image Registration In The DeepPlan Treatment Planning System

Medical image registration techniques have been widely applied to radiotherapy.Traditional methods of medical image registration are based on the image intensity,and this method is knowm to be difficult in finding suitable metric and in performing the task efficiently.On the other hand,the deep learning technology has been found recently to be suitable for medical image registration owing to its ability in identifying inherent and hard-to-discover features from the training data.Although researchers have recently applied the deep learning technology to medical image registration,several challenges remain:(1)Results of image registration that are based on supervised methods are affected by training targets that are hard to obtain from traditional methods.(2)The unsupervised neural network is limited to the one-way network,thus unable to achieve the satisfactory registration accuracy.(3)There is no unified index to evaluate the performance of registration results.This dissertation reports the development of a new deep learning based image registration method,and its application to clinical data through the DeepPlan treatment planning system.Specifically,the research goal is to build a three-dimensional medical image registration software tool that is based on deep neural networks and a medical image visualization module that displays the registration results.To achieve the goal of the PhD project,three tasks are performed:(1)To improve the full convolutional network(FCN)and the adversarial networks(GANs)by adding a contour loss function,the Modality Independent Neighbourhood Descriptor loss function(MIND),and a cycle-consistent method for three-dimensional(3D)medical image rigid registration and deformable registration,(2)To develop a 3D visualization software tool to display the resulting registration images,organ countour images and digitally reconstructured radiographs based on OpenGL and to solve the connection problem of multi-connected regions of organ contours by combining contour connecting method and isosurface method.(3)To integrate the registration software tool and visualization software tool with DeepPlan.Firstly,the unsupervised rigid registration networks based on the contour loss function and MIND are developed using Python and Pytorch programming environment.Then the unsupervised deformable registration networks are developed by introducing the cycle-consistent method and by using cycle loss to improve the accuracy and stability of deformation registration.To display the images before and after the registration,3D visualization algorithms based on CT values and contour information are proposed using the OpenGL graphics library.Finally,the interfaces of registration and visualization software toolsare developed and integrated into the treatment planning system,DeepPlan.A total of 74 cases of MR-CT pelvic images and 98 cases of CT-CT head and neck images are used as training data sets,and the Dice similarity coefficient(DSC)and average surface distance(ASD)of ROIs are calculated as the evaluation index,all ROIs are outlined by physicians from the Department of Radiation Oncology of the First Affiliated Hospital of the Anhui Medical University.For multi-modal rigid registration,the average Dice coefficient and ASD value of the ROIs before registration are found to be 0.398 and 13.93mm,and the average Dice coefficient and ASD value after registration by Elastix and the proposed method are found to be 0.748,6.87mm and 0.753,6.49mm respectively.The average registration time for Elastix is 13s,while the proposed method is less than 0.1s.For multi-modal deformation registration,the average Dice coefficient and ASD value of the ROIs before registration are found to be 0.433 and 14.19mm,respectively,and the average Dice coefficient and ASD value after registration by Elastix and the proposed method are 0.761,4.00mm and 0.803,2.66mm,respectively.The average registration time for Elastix is 64s,while the proposed method is less than 0.1s.These results show that the proposed method maintain the similar rigid registration accuracy as Elastix software,while greatly reduce the registration time.For deformable registration,the proposed method is able to improve the registration accuracy as well as the registration speed.In addition,a 3D visualization tool for medical images based on CT values and contour information is developed to display the registration results,as well as 3D images.The interfaces of registration program and visualization program are developed and successfully integrated into the treatment planning system,DeepPlan.The main innovations of this work are:(1)Developed the unsupervised rigid registration networks based on the contour loss function and MIND,proposed the unsupervised FCNs based on cycle-consistent method for deformable registration,improved the registration accuracy and reduce the registration time compared to common methods.(2)Proposed a new method to combine contour stitching method and isosurface method to solve the connection problem of multi connected regions of organ contours.