Research On Guide Wire Feeding Strategy Of Vascular Interventional Robot Based On Deep Reinforcement Learning

Compared with traditional vascular surgery,vascular interventional surgery has less complications,small wound and high safety.It is an effective means to treat cardiovascular and cerebrovascular diseases.But doctors who stay in the operating room for a long time during interventional surgery will be injured by X-ray radiation,and wearing heavy lead clothes will damage the doctor’s bone system.In recent years,great progress has been made in the research of interventional surgery robot,but its automatic wire feeding function has not been realized.This paper proposes to use the deep reinforcement learning method to study the wire feeding strategy of the vascular interventional robot,and complete the wire feeding task of the interventional surgery by the trained reinforcement learning model,so as to achieve the automatic wire feeding function of the interventional surgery robot.Firstly,according to the analysis of the information required by the doctor for wire feeding in interventional surgery,it is determined to use the guide wire tip as the state feedback of the deep reinforcement learning model.The target detection model YOLOv5s is trained and tested with the made vascular guide wire image data set.The accuracy of the model is 100%on the verification set and 92.9%on the test set with different illumination and pixels,and the accuracy was 98.4%on the real interventional contrast image data set.YOLOv5s model can obtain the position and orientation angle of the guide wire tip in the blood vessel.The orientation angle information of the guide wire tip can be extracted by convolution neural network,so as to complete the state feedback process of reinforcement learning model.Secondly,through the analysis of the wire feeding process of the interventional surgery,the corresponding deep reinforcement learning model is established,the network components of the deep reinforcement learning are built using the SAC framework,and the residual network and the priority replay mechanism are combined to improve the information transmission and data sampling efficiency in the network model.The inverted pendulum game environment provided by open AI gym is used to verify the effectiveness of the reinforcement learning model.The experimental results show that the improved SAC model can solve the inverted pendulum problem well,which provides a guarantee for the subsequent experimental verification on the interventional surgery experimental platform.Then,contrasting the offline reinforcement learning process,the ways in which demo data are combined with the reinforcement learning model are discussed.To examine the effect of the combination of demo data and the reinforcement learning model by combining demo data into the improved SAC model for offline training,an offline reinforcement learning model BCQ was trained as a contrast to examine the effectiveness of the offline trained SAC model and also provide assurance for subsequent online training of the SAC model loaded with offline model weights and presentation data.The above model trained offline will be tested on the interventional surgery experimental platform.Finally,a wire feeding robot is designed to realize the wire feeding action for interventional surgery experiments,and a vascular model is purchased for setting up the interventional surgery physical experimental platform,training and testing the above built deep reinforcement learning models.The experimental results show that the improved SAC model,which combines the residual network structure and preferential replay mechanism,completes the training within 4.2 h,and this model can accomplish the wire feeding task with a straight distance of approximately 150 mm from the femoral artery to the renal artery in an average of 14 steps,the success rate was 90%in 10 tests.Meanwhile,the experimental results show that both the offline reinforcement learning model BCQ and the offline trained SAC model can learn the strategy to push the wire forward,their highest success rates for the wire feeding task are 70%and 90%respectively in 10 tests,and the SAC model performs somewhat better than the BCQ model,which indicates that the offline trained SAC model is effective.But the wire feeding actions of both models were more monotonic,and task success or not was correlated with the initial location comparison.Load demo data and the improved SAC model weights that were trained offline 1 million times into the improved SAC model for online training 200 episodes with a training time of approximately 1.4 hours.The experimental results show that the model achieves 100%success rate on the wire feeding task in 10 tests and the average number of steps per round is 11,which is somewhat better compared to the model without combining presentation data and offline training weights,and greatly reduces training time.Thus,it is known that combining demo data greatly improves the training speed of the reinforced learning model and slightly improves the model performance.