Automatic Establishment And Real-time Tracking Of Human 3D Meridians Based On Deep Learning

In recent years,the use of AI and robotics techniques to empower the modernization of traditional Chinese medicine(TCM)has attracted more and more attention.Real-time identification of TCM meridians using Computer Vision technology is an inevitable key issue in the modernisation of TCM.In this paper,a deep learning-based method for automatic establishment and real-time tracking of 3D meridian is designed and implemented.Based on the Dense Pose deep learning model,the 2D human meridian template is mapped to its corresponding 3D meridian model of the sample human body,and the non-rigid point cloud registration algorithm is used to track the 3D point cloud data of the human body collected by the depth camera in real time,and the real-time tracking of the human 3D meridians is achieved by improving the Transformer model.The main contents are as follows:(1)A deep learning-based 3D meridian establishment and real-time tracking system for the human body was established,which mainly consists of a Microsoft Azure Kinect depth camera and a deep learning workstation.(2)A 2D meridian template of the human body was extracted from the TCM software and transformed into a texture map using 3DMax software.The 2D meridian template of the human body was then automatically mapped into its corresponding 3D meridian model of the sample human body using the Dense Pose deep learning model,thus enabling the automatic creation of a 3D meridian model of the human body.(3)A non-rigid point cloud registration algorithm is designed and implemented based on the probabilistic model registration method to track3 D human point cloud data in real time.The algorithm is based on the Gaussian mixture model,introduces the motion coherence theory,and uses the variational Bayesian inference method to calculate the maximum posterior estimation of the point set,so as to realize the non-rigid point cloud registration process.Due to the computational complexity of solving the non-rigid transformation in the algorithm,the point cloud was downsampled and interpolated,and the intermediate process parameters were calculated approximately,which effectively reduced the complexity of the algorithm.The experimental results show that the improved algorithm not only significantly reduces the computation time,but also the registration accuracy is still considerable.However,the run time of this modified algorithm exceeds 32 milliseconds,and still cannot meet the requirements of real-time tracking of 3D human point cloud because the human real-time3 D point cloud data is too large.(4)In order to accurately and effectively track 3D human body point cloud data in real time,a non-rigid point cloud registration algorithm based on improved Transformer is implemented.Aiming at the scale difference between point cloud and transformation vector,the Transformer network structure is improved by referring to the high-dimensional input extension architecture.Because the attention scoring function of point cloud needs to deal with missing anomalies,the point density weighting method is adopted in this paper.The experiments of tracking human body 3D point clouds and meridian show that the non-rigid point cloud registration algorithm based on improved Transformer model can dramatically improves the speed of point cloud registration,and its average registration time of 3D human body point clouds is 0.3 milliseconds and enable the real-time tracking of human body 3D meridians.