Design And Implementation Of Government Network Public Opinion Analysis System Based On Spark

With the continuous development of Internet technology,a large amount of network information is generated in the Internet.As an important information dissemination platform,people can quickly browse webpage information and share information resources to achieve extensive communication through network events which we can guide the government department to take appropriate decisions.However,with the explosive growth of network data,the performance of traditional public opinion analysis technology is difficult to meet the processing requirements of large-scale data.In order to improve the performance requirements of public opinion analysis,this paper will use the Spark computing platform to process large-scale public opinion data,and combine it with the natural language processing and data mining technologies to improve the efficiency of processing network data and provide timely and accurate network public opinion information for relevant government.The main contents of this paper are as follows:(1)The paper first analyzes the shortcomings of the traditional public opinion analysis system in computing power.A lot of public opinion data is generated on the Internet every day.Iit is first necessary to turn the data into a computer-processable form before analyzing the them.The traditional vector space model consumes a lot of time and space for text representation,which is serious blocked real-time topic detection.In this paper,traditional vector space model representation is used for text parallel vectorization,and sparse vector is used to reduce storage space and computational overhead.(2)In order to solve the shortcomings of traditional sentiment analysis in computing ability and improve the processing efficiency of text analysis.This paper compares the traditional public opinion analysis algorithm to choose appropriate and effective public opinion analysis algorithm for text analysis.Then we parallelizes these algorithms to achieve the purpose of improving calculation efficiency.Spark technology is based on memory operations,and Spark MLlib provides a distributed implementation of massive data machine learning methods,so it is ideal for machine learning which requires iterative operations.This can implement machine learning processes for large-scale data.(3)For the analysis and processing requirements of large-scale text data.We select the technology that meets the system requirements of this paper by comparing technical solutions.Then we design and implement the network public opinion analysis system with Spark as the computing platform to improve the performance of the system.Specifically,the design and analysis of the public opinion data are considered.At the same time,the detailed design and implementation scheme of the public opinion analysis system for the government network is given.Finally,some system interfaces are shown.