Research On Key Techniques Of Vehicle Search In Urban Video Surveillance Networks

Vehicles have become an indispensable part of human life as well as a significant class of objects in urban surveillance systems.Many researchers in multimedia and computer vision fields have focused on vehicle-related re-search,such as detection,tracking,fine-grained categorization,and license plate recognition.Nevertheless,vehicle search or re-identification,which can find the same vehicle in large-scale surveillance videos with a given query,is an important but frontier area.Through the ubiquitous surveillance network,ve-hicle search can quickly tell users where and when the vehicle was in the city.However,the main challenge is how to guarantee both of accuracy and effi-ciency in vehicle search.On one hand,the variety and similarity of vehicles in uncertain environments make it difficult to match the same vehicle accurately.On the other hand,efficient vehicle search is a challenging task due to the huge volume of surveillance data and complexity of vehicle features.This thesis presents a multi-modal and progressive vehicle search frame-work for large-scale urban surveillance scenes.In particular,we propose a se-ries of models and approaches from three aspects of vehicle search:vehicle appearance feature learning and representation,license plate verification based vehicle matching,and vehicle search re-ranking based on spatiotemporal infor-mation in video surveillance networks.Furthermore,through extensive exper-iments on a large-scale dataset collected from real-world surveillance system,we validate the accuracy and efficiency of the proposed framework and meth-ods for vehicle search.The main contributions of this thesis are as follows:(1)Multi-modal and progressive vehicle search framework.The proposed framework utilizes the multi-modal data in large-scale video surveillance net-works to achieve progressive search in both feature domain and spatiotemporal domain.Specifically,the progressive search process includes two aspects.One is the coarse-to-fine search in the feature space,i.e.,first obtaining similar ve-hicles with appearance features,then matching the target vehicles with license plates.The other is the near-to-distant search with spatiotemporal information in surveillance networks.Extensive experiments demonstrate that our frame-work can achieve accurate vehicle search and reduce the time cost through the progressive search manner.(2)Appearance feature based similar vehicle search.We propose two novel deep CNN-based approaches,i.e.,NuFACT and CAN,for single-shot and video-based vehicle search,respectively.For single-shot vehicle search,the NuFACT can extract multi-level appearance features from vehicle images.Then a null space-based metric learning method is adopted to fuse these features as a discriminative and robust representation.For video-based vehicle search,the CAN can learn common features and complementary features from multiple frames in vehicle videos.Then the significant features from images of differ-ent viewpoints and resolutions can be enhanced by the attention mechanism to make the fused features more separable in the feature space.(3)Accurate vehicle search by combining license plate super-resolution and verification.To overcome the low quality of license plates captured in un-constrained surveillance scenes,a domain prior generative adversarial network for license plate super-resolution(SR)is designed to generate a high-resolution plate from a low-resolution one.Moreo-ver,since there are large numbers of vehicles but small numbers of samples for each vehicle,a Siamese Neural Net-work(SNN)based plate verification method is proposed for accurate plate veri-fication instead of recognition.By integrating license plate SR and verification,the accuracy of vehicle search is significantly improved.(4)Search result re-ranking based on camera neighboring graph and spa-tiotemporal similarity.To achieve near-to-far search in physical space,con-textual information,such as the timestamps,locations of cameras,distances between neighboring cameras,is exploited to build a neighboring graph for rep-resenting the topology of the surveillance network.Furthermore,we propose a multi-layer perceptron based the spatiotemporal similarity model(STSM)to estimate the spatiotemporal similarity between two vehicles.Finally,the results of appearance and license plate-based vehicle search are re-ranked to obtain the optimized results.To validate the proposed framework and methods,we build a prototype system of multi-modal data based progressive vehicle search.Extensive exper-iments on data from real video surveillance system demonstrate that the pro-posed progressive framework can find the target vehicle accurately and effi-ciently.