The Establishment And Application Of Automatic Keyframe Detection System And Laparoscope Handing Skill Assessment System Based On Laparoscopic Surgery Images

Objective It is of great significance to establish a robotic automatic laparoscope handing system in the period of laparoscopic surgery.To provide a theoretical and practical basis for the research of automatic laparoscope handing system in laparoscopic surgery,this study was aimed to propose a method for automatic laproscopic critical actions recognition and to establish an automatic keyframe detection system based on laparoscopic surgery images.It was also designed to establish an automatic laparoscope handing skill assessment system and to achieve the goal of quality evaluation for automatic laparoscope handing based on laparoscopic surgery images.Methods(1)Twelve videos of laparoscopic cholecystectomy were adopted.The keyframes were marked by 3 surgeons based on the criteria of critical actions in laparoscopic surgery.At the same time,the classification of instruments and the tips of instruments were marked.The instruments in keyframes were semantically segmented by experts in the field of computer vision recognition.(2)The information of the classification of instrument,the position of instrument,the velocity of instrument moving,the distance from the camera of instrument,the tip of instrument and smoke were selected as the explainable features.The labeled data of the classification of instruments in the database were sent to the neural network of YOLO_v5for training to achieve the goal of automatic detection of the classification of instruments.According to the bounding box,the explainable features of position,velocity,and distance were obtained.The labeled data of the tip of instruments in the database were sent to another neural network of YOLO_v5 for training to achieve the goal of automatic detection of the tip of instruments.The information of smoke was obtained by calculating the image brightness.A two-stage network system was established after then.A complete video of laparoscopic cholecystectomy was fed into the system for automatic detection to evaluate the performance of it.(3)The kernel density estimation model of position,the kernel density estimation model of velocity,the kernel density estimation model of distance,the hidden Markov model of tip and environment detection model were established based on the statistical fitting of the explainable features from laparoscopic surgery images.These models were used to distinguish their own spatio-temporal distribution of keyframes and conventional laparoscopic images.A hierarchical identification system was established in this way.The12 vedios of laparoscopic cholecystectomy were fed into the system for self-verifying.Another 11 unlabeled videos of laparoscopic cholecystectomy were fed into the system for automatic detection.The results were compared with the judgment of 3 surgeons to evaluate the performance of the system.(4)The scale of the global assecement of laparoscope handing skill was designed at first.Six surgeons evaluated the laparoscope handing skill in 300 videos of 20 seconds before and after keyframes based on the scale.Nine classical machine learning regression methods were used as score prediction models of the assecement of laparoscope handing skill to establish the automatic laparoscope handing skill assessment system,which was based on the sufficient statistics of explainable features and the scores of 6 surgeons.One hundred and fifty videos were used for the training,and the other 150 videos were used for verification.The scores predicted by the automatic assessment system were compared with the scores of6 surgeons in the same 150 videos of verification to evaluate the effectiveness of the automatic laparoscope handing skill assessment system.Results(1)The labeling process was successful.The dataset included the data of 12 videos of laparoscopic cholecystectomy,the labeled data of 315 keyframes,the labeled data of instrument classification,the labeled data of instrument tip,and the semantic segmentation information of instruments in keyframes.(2)A two-stage network system was successfully designed to detect and extract explainable features in laparoscopic surgery images automatically.The mean average precision of automatic detection of instrument classification and instrument tip was 0.9954 and 0.9948,and the detection time of perframe was much less than minimum image sampling interval time.The results indicated that the accuracy and real-time performance of the system were good.(3)A hierarchical identification system was successfully designed to automatic detection of keyframes in laparoscopic surgery images,which indicated that a method for automatic laproscopic critical actions recognition in operations was proposed.Six hundred and eight keyframes in 12 videos of laparoscopic cholecystectomy were detected by the system,which was much higher than 315 frames marked manually.Compared with the decision of 3surgeons,the accuracy of the hierarchical identification system for keyframe detection was0.9156,0.9123 and 0.9903,which denoted the effectiveness of the hierarchical identification system for keyframe detection.The average reliability was 0.7868,indicating that the hierarchical identification system was comparable to the decision of the surgeon.The average agreement was 0.6839,indicating that 3 surgeons had similar judgment criteria for keyframes.(4)The Cronbach’s α of the six elements of visual field stability,visual field accuracy,visual field clarity,perspective selection,operation efficiency and total score evaluated by 6surgeons in the scale were 0.7453,0.8105,0.7379,0.7163,0.8757 and 0.8225.It standed for the effectiveness of the scale of the global assecement of laparoscope handing skill.The error percentage of the prediction model trained by 9 kinds of machine learning schemes and the comprehensive evaluation of the 150 videos by 6 surgeons was below 10%,which indicated the effectiveness of the automatic laparoscope handing skill assessment system.The system could be used as an optimization standard for automatic laparoscope handing.ConclusionBased on the criteria of critical actions in laparoscopic surgery,a specific database was established and an explainable features extraction system from laparoscopic surgery images was established,from which basic data and quantitative data for subsequent research could be obtained.Based on the obtained data,a method for automatic laproscopic critical actions recognition in the procedure of operation was proposed and an automatic keyframe detection system was established.The achievement was the answere of how to determine the laparoscope handing time and laid a foundation for the improvement of the automatic control scheme of laparoscope handing.The automatic laparoscope handing skill assessment system was also established in this work,which provided a standard for the optimization of laparoscope handing skill.The achievement was the answere of how to judge the quality of laparoscope handing.