Visualization Analysis Of Big Data Based On Spark

With the surge of data volume in this big data and information intelligence departments and other agencies have stored large amounts of structured and semi-structured data through years of on information accumulation,but different degrees of information barriers and low information sharing degree are generated due to limitation of systematic mechanism,innovation consciousness,support guarantee and many other factors.Therefore,it is still necessary to research a great deal of knowledge that how to use these data to master social trends and analyze evolution tendency.so as to warn in advance and propose decisional recommends for leaders.Big data technology has already tended to be mature after many years of development and improvement,and it can propose reliable guidelines for policy makers by analyzing intelligence through large data and taking advantage of data efficiently.In addition,it can vigorously promote information construction of intelligence departments through information collection,integration and supplemented with data technology.It can provide users with efficient and interactive queries and calculations for rapidly showing data information and improve poeple's working efficiency by using features of Spark running in memory.Based on big data technology,the system,towards intelligence information base,political-legal resources and existed database or other various data files,would provide decision makers with analysis of community groups,analysis of personnel status,correlated personnel queries and other various functions through rapid and efficient information query and diversified graphical displays,supplemented by GraphX calculating framework.The main contents of this paper are as follows:1.Research intelligence system business model,research intelligence,special needs analysis of special departments,Research Spark.Hadoop large data analysis and storage technology,Based on J2EE front-end display system,Distributed messaging systems and data cleansing and database usage,And according to the study,the research results of the design of a large data analysis system to achieve a method.2.The front of the system adopted J2EEtechnical architecture,equipped with three frameworks of Spring,SpringMVC and Mybatis,providing front-end display system of good scalability,maintainability and low coupling.There into,the view layer adopted FreeMarker,JQuery easy UI,ECharts and other component for providing various,simple and visual data display.Besides,it was also carried Oracle database,providing persistent operations for side component,user name,password and other information.3.Distributed computing and transmission system development.Front-end display system would make real-time interaction with Spark clusters through Apache Kafka clusters.Spark clusters are mainly responsible for calculation and analysis of data;implement community discovery,analysis of key personnel,bill analysis and other functions through reading data on the storage system and using Spark RDD,Spark SQL,GrpahX and other tools;then transfer results to the front-end display system through real-time feedback of Kafka clusters.Users can intuitively view the internal information of the data instead of caring about data mining or waiting for long hours.4.Data cleaning and import work.Mass data of intelligence agencies uses distributed storage systems,which support a variety of data sources,such as relational databases,text files and CSV documents,stored in the HDFS system uniformly after data cleaning.Update the storage system through Sqoop,and import the external data into the storage system in a fixed time to ensure the effectiveness of real-time data.Use graph-data processing algorithms for discovering internal relations within crowds.Firstly,abstract people,thing and other information in the real world as the node of the graph,and construct graph data through abstracting relations among things as edges.Look for community and key personnel in the data through LPA algorithms and PageRank algorithms,based on which analyze phone records,travel records,activity status,and other information about community staff.Through GraphFrame computing framework,discover different communities and key personnel in each community among crowds,as well as show people's phone records,duration,number and movements and so on.Big data analysis system achieved information exchange among various departments through collect and extract information scattered in different business sectors,which would break down information barriers as well as provide efficient and intuitive data visualized process,playing an active role for improvement of efficiency and capacity of public security organs.