Research On Key Technologies Of Massive Data Access Take The Big Data Analysis Of Public Security As An Example

On the one hand,with the construction of the “Skynet” of the public security,the construction and improvement of the video surveillance infrastructure,the monitoring video data has seen a spurt of growth,on the other hand,with the new generation of information technology such as big data,cloud computing,and the Internet of Things.With rapid development,the public security department has built a data mining information platform based on big data technology for intelligence analysis and non-contact crime analysis and forecasting.The rapid storage and efficient retrieval of massive video data is the basis for other functions.Therefore,the research and application of efficient storage and fast access technology of massive image data has great practical significance.With the spurt of surveillance video,the amount of pictures has reached more than PB and is constantly expanding.Hadoop's distributed file system HDFS,distributed parallel computing framework MapReduce and HBase provide a good platform for video data processing.Hadoop provides a fast and efficient solution for the retrieval of massive video images.This paper introduces a series of processes for massive video data processing such as face recognition,feature value extraction,file merge transmission,and distributed storage.The paper introduces the data source and data format of public security data and designs the RowKey and column clusters in combination with the requirements of the public security project itself.There are many requirements for non-primary key query in public security big data.In order to further improve retrieval efficiency,the HBase secondary index scheme is given in the paper.In this paper,the local server is used to extract the image feature values,and the image feature values are clustered under the Hadoop platform.Finally,the similar images are mapped into the same hash bucket by using the LSH algorithm.At the same time,the text has made appropriate improvements to k-means.Firstly,the image feature values are randomly sampled to reduce the amount of calculation,and then the density-based clustering process is performed on the data to dynamically determine the cluster center of the k-means clustering algorithm.The K-dist map is drawn from small to large,and the cluster center is searched using the k-dist graph,thereby reducing the number of iterations of the k-means algorithm.Through comparison with traditional storage methods and traditional k-means,it is proved that the k-means algorithm and HBase table design of this paper greatly improve the retrieval efficiency of images.