Research On Object Detection Based On Event Stream

Traditional cameras may suffer from motion blur,overexposure and underexposure in high-speed and high-mobility scenarios or extreme lighting conditions.The low-quality images generated by traditional cameras under adverse lighting conditions can render object detection algorithms designed for high-quality images ineffective.As a novel form of neuromorphic vision sensor,event cameras offer numerous potential advantages over standard cameras,such as high dynamic range,high temporal resolution,low latency low bandwidth and low power consumption.The exceptional properties of event cameras provide new approaches for object detection under adverse lighting conditions.The main purpose of this dissertation is to demonstrate the feasibility of using event streams generated by event cameras for object detection,and to investigate how event cameras can be combined with traditional cameras to improve the accuracy,speed,and applicability of object detection models.To achieve these goals,two event-based object detection algorithms were designed based on different representation methods,along with a feature-level fusion object detection algorithm for event stream and image frame data.These algorithms were validated through experiments on public datasets,demonstrating their effectiveness and practicality.The main contributions of this dissertation include:1.Research on event-based convolutional neural network for object detection.Considering the similarity between event stream data and traditional image data,this dissertation designs an event-based convolutional neural network for object detection based on the idea that "event points in event stream data can be represented as pixels in a two-dimensional image".The algorithm first converts the event stream data into event frames through an adaptive time-quantity sampling method to solve the problem of balancing the image quality of event frames and the processing speed of the algorithm.Then,it extracts semantic features of event frames through convolutional neural networks and predicts the possible object categories and their positions hidden in the event stream data based on the extracted features.2.Research on event-based object detection algorithm using Graph Neural Network.In order to fully utilize the asynchronous,unordered,and high-frequency characteristics of event-based data,this dissertation proposes a novel event-based object detection algorithm using graph convolutional neural network.Starting from the idea that "an event point in the event-based data corresponds to a vertex in the spatiotemporal graph",the algorithm represents the event stream data as a discrete event graph through a radius neighborhood strategy.Then,based on graph convolutional neural network,the algorithm extracts hidden attributes of the graph and predicts the objects that are implicitly contained in the spatiotemporal graph through operations such as graph convolution and pooling.This dissertation also designs an asynchronous updating mechanism for graph neural network,which renders the model to asynchronously update,convolve,and pool the graphs while maintaining high detection accuracy,and reduces the computational complexity.3.Research on object detection algorithm based on the fusion of event stream and image frame features.The dissertation focuses on developing an object detection algorithm that combines the advantages of event cameras and traditional cameras to improve the accuracy,speed,and applicability of the model.To address the problem of information loss or insufficient fusion in existing multimodal fusion methods,we propose an object detection model based on the fusion of event stream and image frame features.This algorithm first represents the event stream data as event frames,and then uses both event frames and image frames as inputs.Through a convolutional neural network,it learns to extract features from both types of frames and fuses the features of these two modalities at the feature-level.Finally,the algorithm predicts the category and location of the object from the high-level semantic information based on the fused feature map.All the three models designed in this dissertation are compared with other algorithms through experiments on public datasets.The results proves the feasibility of using event stream data for object detection and the potential to improve object detection accuracy,speed,and applicability by combining event cameras with traditional cameras.