Research On Object Tracking Technology Based On Event-based Sensor

At present,the image sensor based on the "frame" and "exposure" systems is still used as the main imaging device in the object tracking field.However,traditional image sensors have problems such as low information value density,large power consumption,small dynamic range,low time resolution and motion blur,etc.These problems seriously affect the efficiency of object tracking photoelectric equipment and the practical application effect of related object tracking algorithms.In order to solve the bottleneck problems in the field of object tracking,an event-based sensor with a new imaging system came into being.The event-based sensor is completely different from the traditional frame-based sensor.Instead of capturing an image at a fixed frame rate,each pixel asynchronously responds where the light intensity changes,and then the image data is output in the form of an asynchronous event stream.The output of the event-based sensor is called spatiotemporal data stream or spatiotemporal event stream,which contains position,microsecond timestamp,and polarity information.The event-based sensor has outstanding properties compared to the traditional frame-based sensor with better than120 d B dynamic range,μs-level temporal resolution,milliwatt-level ultra-low power consumption,no motion blur,and good intelligent processing interface,etc.Hence,event-based sensors have considerable potential,especially in high-speed and ultrahigh-speed object tracking scenarios that are challenging for traditional image sensors that are challenging for traditional cameras.Due to the asynchronous spatiotemporal event stream data format used by the event-based sensor,existing target tracking algorithms based on the "frame" image format can’t be directly applied to the event-based sensor.Existing object tracking algorithms based on event-based sensors often slice and integrate asynchronous spatiotemporal data streams into "image frames",and then use traditional image processing algorithms to track objects.This method is similar to traditional imaging systems and is relatively convenient in application mode.However,the spatiotemporal data of the event-based sensor does not contain the grayscale value of the target,it is not possible to accurately track the target in the "image frame" using the color and texture information of the target.Besides,this method of integrating spatiotemporal data stream to construct image frames will cause the loss of object information or the existence of motion blur due to the problem of frame construction,which will affect the result of object tracking.Meanwhile,this method also loses the advantage of low data volume and high spatiotemporal resolution of event-based sensors.In order to solve the above problems,this paper carries out research on object tracking algorithms based on asynchronous event stream data by studying the imaging mechanism based on the event-based sensor and combining the characteristics of spatiotemporal data.First,we complete the adaptive slicing of the spatiotemporal data stream through the research of preprocessing algorithm based on spatiotemporal event stream data.There is neither object information loss nor motion blurs in the spatiotemporal event slice used for object tracking.Secondly,an optical flow prediction network for target tracking applications based on asynchronous event flow data is proposed.The events in the adaptive slice are expressed in discrete form,we predict the optical information for the event which is used for subsequent object tracking,and improve the accuracy of the object tracking algorithm.Finally,through the research of object tracking algorithms and performance evaluation based on the characteristics of spatiotemporal data stream optical flow,object tracking in dynamic scenes is completed.Specifically,the main research contents of this paper are as follows:(1)Carry out the research on the imaging mechanism of the event-based sensor,the detection mechanism of large dynamic,low data rate,and low power consumption of event-based sensors is analyzed and verified.We summarize the existing spatiotemporal data processing methods and lay a foundation for the subsequent spatiotemporal event preprocessing of object tracking algorithms.(2)Carry out the research on the preprocessing algorithm of the spatiotemporal data stream.Aiming at the phenomenon of motion blur or information loss existing in the existing spatiotemporal event slicing methods,an adaptive slicing method is proposed to slice the spatiotemporal event stream.Each slice contains both complete object information and no motion blur phenomenon.Through comparison of actual slicing effects and analysis of indicator calculations,it is shown that the quality of event slicing obtained by this method is superior to other methods.The preprocessing algorithm provides a good input data basis for subsequent target tracking.(3)Research on the optical flow prediction algorithm of the spatiotemporal event stream.In order to solve the problem that the existing optical flow prediction network based on convolution has a large amount of computation and cannot directly deal with discrete space-time events.We propose a spatiotemporal event stream optical flow prediction method based on the spike neural network.The network can directly process discrete spatiotemporal data streams,and it is no longer necessary to construct spatiotemporal data streams into image frames,which greatly reduces the amount of data to be calculated,improves the calculation speed,and retains the advantage of the low data volume of event triggered detectors.Comparative experiments show that the implementation power consumption of this algorithm is about 99% lower than that of traditional optical flow prediction networks based on convolution.(4)Research on object tracking algorithm based on optical flow characteristics of spatiotemporal data.On the basis of the previous research contents,the optical flow field is innovatively introduced into the clustering algorithm to expand the event information dimension and the result of clustering and segmentation based on the spatiotemporal event optical flow field is used as the Kalman filter correction parameter to complete the object tracking and performance evaluation in the dynamic scene.The experimental results on public datasets indicate that the algorithm proposed in this paper is more suitable for tracking moving targets in complex moving backgrounds due to the fusion of optical flow information in the event domain,and has high practical application value.