Interactive Medical Image Segmentation With Multi-agent Deep Reinforcement Learning

Medical image segmentation is widely regarded as the most important step in subsequent medical image processing.Whereas,manual medical image annotation is very expensive.On the one hand,medical images are mostly 3D,which requires a lot of time resources.On the other hand,annotation requires professional doctors and the accuracy is closely related to the doctor’s experience.With the rapid development of convolutional neural networks in recent years,automatic segmentation has greatly improved the efficiency of medical image segmentation.However,for the pratical clinic use,the accuracy and robustness of the existing automatic methods still need to be improved.In order to obtain a better segmentation result,the interactive image segmentation strategy introduces a few user hints,which has become a valuable research direction.In order to reduce the hint number and interaction time,the existing interactive image segmentation methods replace the initial user hints with the automatically-obtained coarse segmentation results.This paper explores and researches on this type of framework.Although these methods can iteratively refine the segmentation results in multiple rounds,they still consider each segmentation update in isolation,largely ignoring the dynamic process nature of successive interactions.In order to make better use of the dynamic process nature in interactive image segmentation,this paper proposes an interactive medical image segmentation method Ite R-MRL based on deep reinforcement learning(DRL).The dynamic process of interactive segmentation is modeled as a Markov decision process which is solved by DRL.Our method considers a segmentation update sequence as a whole,and fully explores the correlation between successive interactive segmentations.Unfortunately,it is intractable to use single-agent DRL for voxel-wise prediction due to the large exploration space.To reduce the exploration space to a tractable size,we treat each voxel as an agnet with a shared voxel-level behavior so that it can be solved with multi-agent RL.An additional advantage of this multi-agent model is to capture the dependency among voxels for segmentation task.For the state,action and reward in RL,a specific design is made for interactive image segmentation task.1)The state design including the segmentation probability retains rich information of previous segmentation and avoids the fluctuation phenomenon of segmentations.2)The multi-scale action design allows the model to make more accurate and finer adjustments to the segmentation probability.3)The reward design provides a comparable baseline for the model and improves its exploration efficiency.Experimental results show that Ite R-MRL significantly outperforms existing state-of-the-art methods on three 3D medical image datasets.Our method requires less interaction and has a faster convergence.In addition,since different simulated user interaction settings will greatly affect the performance of segmentation results,this paper provides an efficient and reasonable proposal of simulation user interaction settings.For various factors of interactive setting,we have done a lot of experiments and in-depth analysis,including whether the user hint point is put on the center or edge of the wrong area,which distance function the hint map generation is based on,the random noise level which is added on user hint point,and the allocation method for clicks given the total number of clicks.