Object Detection And Three-dimensional Positioning Of Hardware Workpieces Under Complex Working Conditions

Several industrial assembly and manufacturing tasks must be accomplished in dark or dimly lit environments with dust,dirt,grime,and grease.These unfavorable conditions make visual recognition tasks under the complex conditions challenging.In addition,because the hardware workpiece belongs to non-Lambert material,it will form reflection spot on the surface under the illumination of light.When the vision system capturing the image,the reflected light spot will form a highlight point in the image,making it difficult to identify the hardware workpiece.Aiming at the problem that hardware workpieces are difficult to identify and locate under complex working conditions,this paper has carried out research on hardware workpiece object recognition and three-dimensional positioning based on Kinect sensors.The main research contents of this article are as follows:(1)The mapping relationship between RGB image and depth image of Kinect sensor is studied,and a method of aligning RGB image and depth image based on Kinect sensor is proposed.This method uses a sparse checkerboard calibration board to calibrate the two cameras,and then obtains the parameter matrices of respective internal and external parameter matrices.Then,the mapping relationship between the RGB image and the depth image is calculated from the internal and external parameter matrices.Through the mapping relationship between the two images,the depth image coordinates corresponding to the RGB image can be obtained.(2)Aiming at the reflective characteristics of hardware workpieces,a backlight illumination scheme was proposed to suppress the reflected light spots on the surface of hardware workpieces.The direct area array LED light source is installed at the bottom of the detection platform,and the contrast between the hardware workpiece and the background is increased by the method of backlight illumination,so as to achieve the purpose of suppressing the reflection of the hardware workpiece.Secondly,an improved Faster-R-CNN deep convolutional neural network is proposed for object detection of hardware workpieces.To improve the network structure of the original Faster-R-CNN deep neural network,the VGG16 network,which is easier to modify and more capable of characterization,is used as a feature extraction network.At the same time,the last two output layers of VGG16 are modified,the feature matrix output by block5 was be up-sampled,and the feature matrices of block3 and block4 are combined to form a multi-scale feature extractor.The feature map generated by the multi-scale feature extractor contains both semantic information and local feature information,thereby improving the model's ability to represent the features of the workpiece.Finally,the edge extraction algorithm is used to extract the edge contour of the workpiece in the prediction frame to obtain the accurate workpiece position information.(3)For the three-dimensional positioning of hardware workpieces,it is transformed into the pose estimation problem of workpieces.By converting the depth image obtained by Kinect sensor into local point cloud data,and fitting the point cloud plane of the point cloud of the local workpiece by Ransac algorithm,the normal vector at the centroid is estimated as the initial pose of the workpiece.Then the initial pose is registered with the CAD template point cloud,and the transformation matrix was obtained.Finally,the voxel-based ICP algorithm is used to refine the registration result to obtain the precise pose of hardware workpiece.(4)A low-cost hardware workpiece detection system based on Kinect sensors was designed,and a hardware workpiece object detection and three-dimensional positioning experiment platform is set up under complex working conditions.The experimental platform was used to verify the positioning accuracy of binocular system for Kinect sensors and the object detection experiments of hardware workpieces.The experimental results show that the system we proposed can meet the identification and positioning requirements of hardware workpieces.