A Gesture Recognition System Based On Dynamic Vision Sensors

Neuromorphic computing mainly aims to simulate the working principle and structure of the biological nervous system through software algorithms or hardware architecture to complete various intelligent tasks.It is an important research field for the realization of artificial intelligence.Neuromorphic computing also proposes to efficiently acquire external information based on neuromorphic devices such as Dynamic Vision Sensor(DVS),and efficiently process impulse information by building brain-inspired algorithms such as Spiking Neural Network(SNN).Neuromorphic processors and related software development greatly support the efficient deployment of biologically inspired algorithms,and synergistically simulation of the functions of biological nervous systems,from the perspective of both hardware and software.DVS is a neuromorphic sensor based on the principle of biological retinal imaging.It is also an important sensing component for neuromorphic computing.DVS is featured a high temporal resolution,high dynamic range,low data redundancy,and high energy efficiency.In addition,as an important computational model for neuromorphic computing,SNN has made remarkable progress in fields such as image recognition,time series prediction,and action detection,and is suitable for edge devices for the low computational and storage overhead.With outstanding advantages in power consumption and scalability,Neuromorphic processor progressively accelerate various SNN applications in both the edge and cloud scenarios.As a vital application in human computer interaction,gesture recognition has been applied widely on many smart and embedded devices.In order to better apply DVS to realize static gesture recognition,extend the application scope of DVS,and realize low-power real-time recognition system,we have conducted following researches:1)Design and record a static gesture dataset based on the event camera Celex IV.We firstly recorded a static gesture dataset of more than 100 k in image resolution.Serval effective event preprocessing algorithms are fully discussed to improve the quality of proposed dataset.2)Design a real-time classification system based on the Celex IV event camera.By proposing a key-frame detection method,the real-time frame rate is improved by 10%.The system with Keyframe method can remain the 30 fps runtime requirements,with competitive real-time recognition accuracy.We also fully explore the CNN model,with system’s accuracy reaching the highest level for the same task(99.749%).By designing sub-window sampling with multi-reservoir structure,the recognition accuracy of SNN model is also improved considerably(94%),with an average enhancement of 3%.Compared to CNN model,the parameter amount of our proposed SNN model decrease considerably by more than 90%.3)Based on Coroutine-based Cosimulation Testbench Environment(COCOTB),a verification platform for Network on Chip(No C)and multi-core neuromorphic processor is proposed.It realizes convenient generation of incentive files,logic verification and so on,for neuromorphic hardware.Based on a FPGA verification platform,we conducted hardware testing for 2D Mesh No C,and No C performance analysis with Vivado simulator.In order to test the neuromorphic processor Unicorn,by applying SNN models,the platform supports functional verification at different granularities,SNN model mapping tests,and preliminary performance analysis.