Remote Collaborative Assembly And Maintenance System Based On HoloLens

With the continuous development of remote office and the continuous promotion of "Industry 4.0",modern production equipment is moving towards the large-scale,automation and integrated development.The equipment is getting increasingly complicated,making the fault diagnosis much more difficult.If there is a fault that can not be found and eliminated in time,it will not only cause damage to the equipment itself,but also cause chain accidents involving the whole department,resulting in huge economic losses.Considering the labor cost,traffic,time and other factors,the industrial industry is more and more urgent for the remote equipment assembly and maintenance system.Based on the fact,this paper proposes a remote collaborative assembly and maintenance system based on the wearable hybrid reality device Holo Lens helmet.Through the Internet,experts with rich professional knowledge and a large number of users with operating experience can cooperate to discover,eliminate and solve various problems of the device in time.In view of the traditional two-dimensional annotation method that without dynamic demonstration or adequate spatial understanding,this paper proposes some three-dimensional spatial annotation methods including directional annotation,sketchy annotation and literal annotation.This paper can provide more visual information for users and experts with the combination of YOLO network and the tool target under the scene of timely detection,assembly and maintenance.The main work of this paper includes the following aspects:1.An action instruction annotation method of directional annotation is proposed,which is used to guide the users’ next operation.In order to improve the interaction experience of experts and avoid manually adjusting the position,size and direction of the model.This paper uses Dollar 1 algorithm to recognize the hand-painted gestures of experts,and improves Dollar 1 algorithm by combining the attributes of hand-painted gestures such as the minimum external rectangle and the timing of hand-painted gestures.According to the gesture information drawn by experts,we can determine the pose of 3D model and anchors it in 3D space.The experimental results shows that the improved Dollar 1 algorithm can improve the interaction experience of experts on the premise of ensuring recognition accuracy and real-time.2.A circle instruction annotation method of sketchy annotation is proposed to emphasize the elements or parts in 3D space.In the process of generating 3D model that composed of the circle contour curve drawn by experts,there are too many folds on the surface of 3D model generated by Teddy algorithm and the algorithm generation time is too long.In the light of these shortcomings,this paper proposes the Teddy algorithm based on Laplacian smoothing,which improves the smoothness of the model surface by adjusting the top point position of the mesh.The experimental results shows that we can reduce the details of the 3D model on the promise of affecting the users’ understanding.In this paper,the Teddy algorithm based on Laplacian smoothing can reduce the number of triangles and its vertexes of 3D model by 60% and it also can reduce the generation time by 78%.3.Based on the YOLO-v3 network model framework and the unity development platform,we realize the real-time detection and recognition of the target in the clients’ work scene.According to the scene of assembly and maintenance,the data set of screwdriver and wrench is made in this paper,which including 1000 images of 7 kinds of screwdrivers and 7 kinds of wrench tools under different light,angle and background conditions.The experimental results shows that the processing capacity of YOLO-v3 can reach 60 fps,which meets the real-time requirements of the system.At the same time,after 2700 iterations of training,we reduce the loss to 0.0171.The YOLO-v3 can accurately identify and detect single target,multi-target and occluded target.4.According to the application requirements of this paper,a remote collaborative assembly maintenance system based on Holo Lens is developed.In order to further improve the user experience,some auxiliary functions such as spatial ranging,automatic assembly of simple parts,3D spatial annotation management are designed and implemented to help users to complete their work more conveniently.The system function is verified in the remote collaborative experiment scenario of printer assembly and auto parts inspection.The experimental results shows that the system can improve the users’ ability to understand the information in the remote collaboration,and improve the work efficiency by 20% with the way of referring to the instruction handbook.