Research On Object Recognition Method Based On Event Data Fusion Of Vision And Touc

Visual and tactile perception are crucial for intelligent robots to complete interactive tasks with the environment,including object recognition,pose estimation,object description.These tasks cannot be completed without intelligent sensors and machine learning algorithms.The binding energy of the two can enable intelligent robots to accurately perceive the environment and make accurate decisions.Event-based sensors can achieve higher energy efficiency and low latency.However,the event data provided by event-driven sensors is based on the "spikes" of events,with high sparsity and spatio-temporal complexity.Therefore,how to fully and effectively utilize visual and tactile event data information for event-driven learning is a research difficulty.Spiking neural networks have rich spatiotemporal dynamics and event-driven characteristics that fit hardware,making it possible to efficiently process event data containing spatiotemporal information.In this paper,we address the problem of difficult extraction and learning of event data features due to its spatio-temporal sparsity,and carry out research work with visual and tactile event data as specific research objects.The main research work is as follows:(1)Aiming at feature extraction of tactile event data,a multi-channel spiking graph convolution tactile event data recognition method is proposed.By constructing a tactile topology map and a feature map for irregular taxels,the spiking graph convolution method is used to extract tactile event data information from the feature information and topology structure of taxels,as well as their combinations.(2)In the problem of gradient error backpropagation training and learning methods that cannot be directly applied in spiking neural networks due to binary transmission signals,the loss gradient related to the time of neuron excitation spike is calculated by considering whether the activity characteristics of spike neurons are triggered during the network propagation process to calculate the influence between neurons and the changes in the internal state of neurons at different times.(3)Aiming at the problem of feature extraction of visual event data,an object recognition method of visual event data based on spiking convolution neural network with regularization method is proposed.Through feature analysis of visual event data,spike input and spike excitation threshold are balanced by spike neuron normalization method,and feature extraction of visual event data is performed by structural regularization and other means to improve model convergence and prevent overfitting.(4)Aiming at the limitations of single visual or tactile perception in object recognition tasks,an object recognition method based on visual and tactile event data fusion is proposed,which avoids the expensive cost of converting discrete event data into real valued tensors and further improves the performance of object recognition based on event data streams.