One-Shot Structured Light Imaging System With A Color Gird Pattern For Real-Time 3D Reconstruction

As an effective means to obtain target spatial topography information,3D imaging technology has received extensive attention in the field of computer vision.Among many 3D imaging methods,structured light imaging has become a research hotspot due to its simple structure and high accuracy of close-up measurement.According to the number of projection patterns used in each measurement,structured light imaging methods can be classified into two types: multi-shot and one-shot.Among them,the application scenarios of the multi-shot methods are mainly concentrated in the industrial offline measurement process that requires high stability and accuracy.Although these methods can provide dense 3D point clouds with high accuracy,the measurement time is long and they cannot capture moving objects;Comparatively,the one-shot methods need a short measurement time and are suitable for realtime application scenarios.For the one-shot structured light imaging,the methods using a color grid pattern based on the De Bruijn sequence have the advantages of fast and accurate decoding,robustness to noise,insensitivity to discontinuities,and applicability to 3D reconstruction both for a static and dynamic scene.However,the drawback of this method is that the number of 3D points is relatively small and their location accuracy is not high,which is caused by the following reasons.(1)The projection pattern has a low resolution and is easily affected by color noise,ambient light,and the texture of the object surface.(2)The camera-projector pair calibration method is not very accurate.(3)The feature point detection method is not efficient enough.(4)Many feature points at the boundary regions of the objects are ignored.And(5)the existing potential risk of false decoding due to the low image quality.In order to improve the quality of point cloud reconstruction in practical applications of the one-shot structured light imaging methods,this thesis conducts in-depth research on imaging system calibration,grid encoding and decoding,feature point detection,and other issues.The main work and contributions of the thesis are as follows:(1)Based on the analysis of the advantages and disadvantages of the traditional grid pattern coding strategies,a new projection pattern is designed which uses five colors to encode the slit on both axes in the HSV color space.Compared to the previous projection pattern,the density of the point cloud is doubled,while color noise and ambient light interference are significantly reduced,allowing it to be used in real-time under indoor lighting conditions.(2)Calibration of a camera-projector structured light imaging system based on an improved color grid pattern.Previous methods of calibrating structured light systems often encountered two problems.(a)Using a printed checkerboard as a calibration object,so the accuracy is not high.And(b)using camera calibration parameters directly calibrates the projector,so camera errors propagate to the projector.Different from them,the proposed calibration method includes two stages: the initial calibration of the camera and the synchronous calibration of the whole system.Firstly,the color grid pattern and blank pattern are projected to the calibration object respectively,and the images of the calibration object at different poses are collected.The camera is then initially calibrated using Zhang’s calibration method.After removing the checkerboard area,the whole system is calibrated using the intersections in the deformed color grid pattern image.Finally,the system parameters are optimized using Bundle Adjustment(BA),and the calibration accuracy is improved through multiple iterations.Thereby,this calibration method can reduce the propagation of camera calibration error to projector calibration,so the system calibration is more accurate.(3)A novel opened-grid-point detection method is proposed to detect feature points,i.e.,intersections in the images of a color grid pattern.In indoor lighting environments,when capturing images of objects with complex textured surfaces,existing methods for 3D reconstruction using a color grid pattern have some drawbacks,such as sensitivity to ambient light and sparse 3D map.The main reason for these limitations comes from the limitations of methods for detecting and decoding feature points.To solve these problems,this thesis introduces a new concept of opened-grid-point to detect feature points,which not only makes the detection of feature points more accurate but also significantly reduces the number of virtual feature points in the regions where the projection pattern is extremely distorted,obtaining fewer but more accurate candidate points,significantly reducing the time required to find correspondence between the camera and the projector images.Additionally,traditional methods only focus on decoding the center point of the 3×3 property window and ignore the points in the object boundary region that do not satisfy the horizontal and vertical codewords.Thus,this thesis proposes a complete decoding method for colored grid patterns with eighteen different codewords,where each feature point can be assigned a specific codeword in the property windows.On this basis,all detected feature points can be decoded correctly.With the openedgrid-point detector and complete decoding algorithm for colored grid patterns,the method proposed in this thesis can acquire a denser point cloud in a shorter time.(4)To improve the quality of 3D maps,a grid-point detection method base on the U-net model is proposed.Where a specific dataset for training and validating the model is designed as follows.First,the color grid pattern is separated into vertical and horizontal color slit patterns.These three patterns are then respectively projected onto the scenes and the images are captured by the camera.The images of a color slit pattern after being preprocessed(denoise,thinning)are fused into a single skeleton image and selected as the label image.This image and the corresponding original image,which is captured with the color grid pattern,are cut into patches and divided into training and validation sets.With the advantage of the U-net in the image segmentation task,our method can significantly eliminate the influence of color noise and ambient light so that clean grid pattern images can be obtained.From these resulting images,the grid points in the images of a deformed grid pattern can be detected with high pixel location accuracy,thereby enhancing the quality of the reconstructed 3D map.(5)A test platform based on two types of the projector,DLP and LCOS,is built to study the effects of different projection technologies on 3D imaging experiments.Based on analyzing the operating principle of the two projectors,the color reproduction performance of the twocolor grid projection patterns captured by the camera is compared.The test results show that the image of the DLP projector is clearer and with higher contrast,but it is greatly affected by color crosstalk,especially in the extremely large distortion region in the images of a deformed pattern;Meanwhile,LCOS projector can produce smoother images and more natural colors.Then,a comparative experiment on 3D reconstruction is performed under the indoor lighting condition and found that: When capturing a static scene,the number of point clouds reconstructed with the DLP projector is less,but higher location accuracy compared to the ones with the LCOS projector;When capturing fast-moving objects,since the DLP projector provides color channels(red,green and blue)of the color grid pattern image in the sequence,the images of the deformed pattern cannot be decoded,so 3D maps cannot be reconstructed.However,the LCOS projector still provides the same high-quality pattern images as capturing static objects,making them more suitable for applications of capturing fast-moving objects.Based on the above methods,a single-shot structured light 3D imaging system using a color grid pattern is built,and the imaging algorithm and system performance are verified through simulation and physical experiments.Comparison of image quality achieved with different color grid patterns and after superimposing different degrees of noise,the novelty and robustness of the proposed color-coding method are verified;In the system calibration stage,the reprojection errors of cameras,projectors,and stereo vision imaging are less than 0.178,0.251,and 0.223 pixels;In the experiments with the objects with different shapes and materials,the correct rate of pattern encoding and decoding is higher than 99.59%;In the standard sphere imaging experiment with a size of 150 mm,the mean error and standard deviation of 3D reconstruction are 0.2795 mm and 0.1507 mm,respectively.To demonstrate the ability to capture fast-moving subjects,the fan rotation speed in the experiment is set to 800 rounds per minute.With the technique of tracking feature points in deformed pattern images from frame to frame assuming no more than 10 mm of movement between two consecutive frames,the system can provide 60 depth maps per second,thus satisfying the requirement of real time 3D reconstruction.