Teaching And Skill Learning For Multimodal Information Fusion

Since the mid-20 th century,the development of technology has greatly promoted global economic growth.In manufacturing,the widespread adoption of automation and robotics has improved the efficiency of assembly tasks.However,traditional automation and robotics technologies are often limited to structured scenarios,and in some highly delicate operations such as the assembly of mobile phone ribbon cables,manual operation is still heavily relied upon.In the context of the aging population trend and the continuous rise of labor costs in China,establishing smarter and more flexible operating methods is crucial for highly refined assembly scenarios.Therefore,this paper proposes a multi-modal information fusion teaching and skill learning method that combines multi-modal fusion technology and deep learning technology to overcome the above problems.The main work is as follows:(1)A multi-modal fusion strategy for the perception of mobile phone ribbon cable assembly environment is proposed,which first preprocesses and aligns the singlechannel information of vision,touch,and speech,analyzes effective feature representation methods for each single-channel information,and then uses the featurelevel data fusion method to fuse multi-modal data,providing multi-channel data support for subsequent skill learning.(2)This paper proposes a skill learning method based on Siamese neural network to solve the problem of the distance between the buckle position and the ribbon cable placement position in mobile phone ribbon cable assembly.This paper collects visual,tactile,and language information for the errors between the ideal and current positions of the ribbon cable,and uses the convolutional neural network Auto Encoder for feature extraction and fusion.Then,the fused features and optimization objectives,which minimize the error between the ideal and current positions,are regressed using a fully connected network.This paper compares this method with classical visual methods and other methods,and the results show that this method can significantly improve the positioning accuracy of mobile phone ribbon cables.(3)Based on the above-mentioned multi-channel information fusion and skill learning methods,a Redmi phone assembly platform is established.Firstly,visual means are used for the preliminary positioning of ribbon cables,then a buckle position and ribbon cable placement alignment strategy based on the Siamese neural network is employed,and finally,the assembly method established in this paper is verified through the traditional spiral search hole buckle strategy for practical ribbon cable assembly performance.Experimental results demonstrate that the proposed multi-modal fusion skill learning method for assembly is more accurate in positioning mobile phone parts than traditional assembly methods and has stronger robustness.