Spark-based Big Data Personnel Intelligence Management System

As an important department for maintaining and protecting social security,the public security bureau and intelligence department have accumulated many different types of databases and stored massive data information for many years.However,because most of them are traditional relational databases,their performance is restricted and the value of these data can not be brought into play.The disadvantages are mainly reflected in the decentralization and lag in the sources of intelligence information,the weak ability to maintain stable information,the degree of automation/visualization of data query and presentation,the low number of proprietary business model applications,and the lack of relevance of data.And so on.How to screen the relationship of these massive data,choose key personnel and analyze their relations network has became an urgent problem that public security agencies need to solve.In this paper,a large data personnel information management system is designed to meet this demand.HDFS distributed file storage system is used together with MySQL relational database to store data.The Spark distributed file calculation engine is used for data analysis and processing,and then the web front end is matched.Technologies such as technology and EChart data visualization,designing a personnel information management system with strong scalability,fast and efficient data processing,and high practicability and simple operation,providing personnel information search,personnel information data visualization,and personnel for public security and intelligence agencies Information data prediction and other functions can be used for public security investigations,public security management,criminal investigation,etc.At present,the system has been put into use by a public security department in Shenzhen and has achieved very good results.The main contents of this paper are as follows:1.Analysis of the development status of big data storage and processing technologyThe current development and research status of the existing big data storage,processing,and personnel information management systems are described in detail.The significance of the topics in this paper is explained.The main technologies used in thesystem design process and the main contents of the research topics are introduced in detail.2.The analysis of overall demand of personnel information management systemAccording to the needs of the use and system construction proposed by the public security and intelligence departments in a district of Shenzhen,the requirements of the personnel information management system are analyzed in detail.It mainly includes the system performance requirements,system data requirements,and the overall functional requirements of the system and the requirements for the construction of a single subsystem.Based on the demand analysis,the research and design method of the system is discussed.3.Data import and screeningThe system adopts distributed data storage and supports relational databases,csv,txt and other data sources to import data.The existing relational database data of the public security department is stored in distributed storage through sqoop.And other types of data are uniformly uploaded and stored in the distributed file system after data cleaning.The system updates data in real time to ensure the effective implementation of the system data.4.Data processing implementation and data processing algorithm analysisThe system makes full use of the existing data,and combined with technologies such as Spark RDD and Spark SQL to implement personnel information query,personnel information statistics,and other functions.Through kafka feedback to the front-end interface for data visualization and interaction feedback.The value of data has been fully utilized to help the public security and intelligence personnel to carry out the research and analysis.The system incorporates machine learning algorithms such as cluster analysis in the MLlib algorithm library to mine deep data values.Combined with spark SQL technology to analyze and process data,this paper introduces,tests and optimizes the algorithms used in the system design process.5.Implementation and testing of personnel information management systemAccording to the demand analysis requirements,the system design is completed and the basic functions to be implemented are tested and improved.Verify and test theimplemented functions and system performance in the system and modify the bugs in the test process.At the end of this paper summarizes that the subject innovation and research results.It also provides an outlook for the research problems and improvements,and lays a good foundation for further research and development of the project.