Low Cost And Latency Processing Of Events For Dynamic Vision Sensor

Dynamic vision sensor(DVS),inspired by bio-retinas,offers a hardware solution to overcome limitations of conventional cameras,with low latency(10μs),low power(10 m W),high dynamic range(120 d B),and high temporal resolution(1 MHz).DVS re-ports the log-intensity changes of each pixel in microseconds with asynchronous events,reducing the redundant output from the background.The high dynamic range and sub-millisecond output of DVS enable vision under poor lighting with quick responses.There-fore,it has enormous advantages and prospects in applications that have limitations on power and latency like video-based surveillance.However,DVS output includes background activity(BA)noise events.These BA events come from pixels even in the absence of any scene activity and are thus nonin-formative.Under dim lighting,BA noise dominates the DVS output.Existing denoising algorithms are not effective under these high noise conditions.Furthermore,it is dif-ficult to quantitatively compare algorithm accuracy.Another problem is that existing frame-based algorithms can not be directly applied to the DVS event stream,since it is asynchronous and has no intensity information.Unlike DNNs for frames,a unified pro-cessing algorithm for this new data format has not been developed yet.Spiking Neural Network(SNN)can process events directly and have low power consumption.However,it also meets challenges in structures and applications.To address these key challenges,in this paper,we focused on developing event pro-cessing algorithms in a low cost and latency manner for DVS BA denoising and two typical surveillance recognition applications.To fairly compare BA denoising methods,we first proposed a novel framework with reconfigurable parameters to better quantify BA denoising algorithms with known mixtures of signal and noise DVS events.Then,we developed three new algorithms that have low cost and latency but good denoising ac-curacy.For event feature extraction,we studied the spiking neural networks(SNN)and proposed a hardware-friendly method for implementing the spiking max-pooling layer.And taking into consideration the ability of DVS to capture motions,we chose fall detec-tion to study the advantages of the combination of SNN and DVS,as well as the infulence with and without noise.Our contributions are as follows:·We propose a quantitative evaluation method for event denoising with reconfig-urable parameters.Our paper contributes detailed observations of DVS noise under low and high light intensities,and we use accurate models of this BA noise to de-velop evaluation method.Our approach is unique by applying problem inversion:Previous work modeled idealized DVS behavior and called anything not ideal as noise;we acknowledge the complexity of DVS pixel dynamics and instead model the measured characteristic of BA noise,so we can then add this simulated noise to clean DVS recordings to see how well we can remove it.We also present the first use of receiver operating characteristics(ROC)and area under the ROC curve(AUC)to characterize denoising,to avoid the shortcomings of the previous work’s arbitrary choice of signal versus noise discrimination threshold.·We propose three low cost and latency BA denoising algorithms.Algorithm 1 uses hash functions to encode event information and store them in a fixed-size corre-lation array.It is memory-efficient and works for low noise level scenarios with stationary cameras.Algorithm 2 checks the distance to past events using a small fixed-size window and removes most of the BA while preserving most of the sig-nal for stationary camera scenarios even at a high noise level.Algorithm 3 is a lightweight multilayer perceptron classifier driven by local event time and polarity surfaces and achieves the best accuracy over all datasets.It is more than 10~4times cheaper than prior machine-learning denoising methods.·We propose a low-cost implementation method for max-pooling layer in spiking neural networks.We only use Integrate-and-Fire(IF)neurons for implementing the spiking max-pooling layer.We show the feasibility of this implementation,by analyzing the spike trains generated by the IF neurons corresponding to three input cases of the pooling window.The experimental results show that SNNs using our method achieve high accuracy and require less inference time than SNNs using existing methods.·We propose an event-based fall detection system.The SNN takes denoised DVS events as input.And by calculating the Class Activation Map(CAM)after SNN inference,we get the central point of the moving object over a fixed time interval.Then a pre-trained machine learning classifier takes a fixed number of continuous central points as input,and predicts whether or not the input stream contains a fall.For evaluating our system,we convert a standard fall detection dataset to its event format and add noise into it.Experimental results show that our system achieves similar accuracy as existing CNN-based methods,and is twice as fast as existing work using DVS for fall detection.Furthermore,high noise levels will degrade the system performance,and with denoising this decrease can be avoided.