3D Object Tracking With Local Optical Flow Supervision

3D object tracking based on RGB data is a fundamental problem in computer vision,and it plays an important role in many application fields.Many popular applications rely on robust and accurate pose tracking algorithms,including robotic perception and manipulation,augmented reality(AR),and human-robot interaction.With the popularity of smart mobile devices and the improvement of computing power,the 3D object tracking method based on monocular color video has become a research hotspot.The task of monocular 3D object tracking is to estimate the pose of a target object relative to the camera in a video.Achieving stable and accurate 3D tracking of textureless or weak-texture objects is a challenging problem.Currently,the 3D object tracking methods with the best performance all use traditional feature extraction methods to implicitly segmenting objects for pose optimization.In extreme cases such as noise,dynamic lighting,and complex backgrounds,it is easy to cause tracking inaccuracy.Definitely data-driven deep learning methods have strong feature extraction capabilities,but unfortunately their accuracy and performance in pose tracking are relatively weak,and often suffer from a trade-off between accuracy and generalization.In 3D object tracking field,the motion features represented by optical flow are ubiquitous and the optical flow itself represents the corresponding relationship between the same-named image points.It is an advanced feature with smoothness and edge mutation,which is helpful for accurate Object segmentation.However,this feature has not been properly exploited in previous 3D object tracking methods.Therefore,we propose a temperal object region segmentation deep learning network LFOS-Net and a traditional pose-optimized tracking framework TMF to achieve robust 3D object tracking.The main research content of this paper is as follows:1.Aiming at the inability of traditional 3D object tracking methods to accurately perceive the non-local feature of textureless or weak-texture object area on the image,a tracking framework TMF which combines an object region segmentation network with traditional region-based pose estimation method is proposed.It combines the robustness of deep learning network feature extraction with the accuracy of traditional optimization methods to enhance the stability of 3D tracking.2.To address the issue of traditional 3D tracking methods being prone to interference and instability due to the complex and variable static features in images,an optical flow segmentation network LFOS-Net is proposed to predict the foreground area of the current frame.LFOS-Net combines the object shape prior and the temporal features of object motion to predict the object mask,and by explicitly introducing optical flow loss,it enhances the network’s reasoning ability for dense correspondence between two frames of images,and improves the robustness and generalization performance of the network.In order to deal with weak motion features,LFOS-Net also utilizes local matching information to segment the foreground area,so that the foreground area on the image can be better estimated for moving objects at different speeds.3.A synthetic dataset for 3D object tracking—RMOT dataset is proposed to train and evaluate the tacking algorithms.It uses 3D models to arrange the close-up view,and uses HDR environment maps as the distant view to set up the scene,taking into account the rendering efficiency and the authenticity of the simulated scene,and to a certain extent bridges the gap between the current rendering data and the real shooting data.The RMOT dataset contains multiple objects,multiple scenes and multiple random motion trajectories,which provides strong support for deep network training.