Research On 3D Reconstruction Of Heart Surface Based On StyleGAN

In robot-assisted minimally invasive heart surgery,3D reconstruction of the scene of the dynamic issue can help enhance the surgical robot’s awareness of the environment and improve the maneuverability of the surgery robot system.At present,due to the influence of a large number of uncertain blood fog,artifacts,mirror reflection and other complex environmental factors in the human body,accurate and stable 3D reconstruction of soft tissue scenes in the wall still face many challenges.Therefore,this thesis studies the dynamic soft tissue 3D reconstruction technology based on a generative network.The main contents are as follows:（1）Aiming at the data set dependence,poor robustness,and other problems of traditional methods,this thesis proposes a new method for disparity estimation of soft tissues from stereo-endoscopic video based on Generative Adversarial Network.First,a decoupled high-rank thin-plate spline model is built to estimate depth information of soft tissues’ recorded video,and then make this depth information as a training dataset to train a simplified StyleGAN Network,Finally,find the Target vector in the latent space of the trained generator,a nonlinear parallax interpolation model.Set left and right images’ consistency loss as the cost function.Finding the most reasonable disparity image matching the left and right images.The performance of the proposed method is finally verified on stereo endoscopic video recorded by the Da Vinci robot,compared with several traditional methods.Experimental results show that the proposed method in this thesis has higher accuracy and better robustness in soft tissue 3D reconstruction.（2）Aiming at the speed problem of the method proposed in（1）,this thesis designed and trained the encoder neural network of the inverse StyleGAN generation network,and proposed a new method of disparity estimation for stereo endoscope images,which is based on the encoder neural network.Firstly,this method generates a training dataset of stereo endoscope images,potential vectors of random disparities and target potential vectors based on this trained StyleGAN generation model,and then design a network that can embed images into StyleGAN’s potential space using the encoder.The encoder network can learn the differences of current disparity and stereo images,and then map it to the vector potential.Finally,the trained encoder network is added as part of the optimizer into the optimization of potential vector retrieval of StyleGAN model,and a reasonable disparity that can match left and right images is obtained.Experimental results show that the proposed method not only significantly improves the retrieval speed of StyleGAN model’s potential vector,but also performs better than the StyleGAN model in accuracy and robustness testing.