Real-time Analysis And System Implementation Of College Network Public Opinion Based On Spark

With the rapid development of network technology,network media plays a more and more important role in people's lives while the Internet has become a huge platform for public information sharing.Internet users express their opinions and opinions via posts,Weibo,forums,etc.where it is unavoidable to have some negative comments that may bring great negative effects and instability to colleges and universities and even the whole society.The real-time analysis system of college Internet public opinion is able to effectively analyze and warn the related public opinion in colleges,which can help colleges and universities to formulate relevant measures to guide the correct direction of public opinion,to stop erroneous statement in time.Therefore,it is of great practical significance to maintain the stability of colleges even the whole society.Prior to this,many scholars have studied and explored the supervision of network public opinion in colleges and universities.However,the relevant researches mostly focus on theoretical and mechanism analysis,lacking practical analysis methods and empirical research,which is difficult to effectively support the actual requirements of colleges and universities under various sudden public opinion.By means of big data,this paper integrates data acquisition,processing,analysis and visualization,and puts forward a real-time analysis method of college network public opinion based on Spark and implements the technology.The real-time monitoring,analysis and early warning of network public opinion in sudden outbreak prevention and control in three universities in China are carried out.The main research contents are as follows:(1)According to the defects of traditional calculation mode in actual text classification engineering application,incremental Bayesian algorithm is improved based on Spark MLlib.The sample size was expanded by fitting and screening the label-free data by the method of sample increment.Then,the new trained samples were used to recalculate the conditionalprobability of each category.Through multiple iterations,the classification accuracy is improved gradually with fewer label samples.(2)We design and implement the Spark-based real-time public opinion analysis framework for university networks,using an improved distributed incremental Bayesian classification algorithm to improve the efficiency of real-time computation and reduce the computation time by investigating the advantages and disadvantages of the existing public opinion analysis framework and system,and modeling the real-time control of public opinion in university networks in the context of big data.(3)With analysis and comparison of the existing distributed learning framework,a real-time analysis system of Internet public opinion based on Spark is designed and implemented as functions of network public opinion data collection,preprocessing,public opinion analysis,visualization,search and early warning functions are completed based on the application background.In this paper,big data technology is adopted to implement distributed incremental naive Bayes algorithm on distributed machine learning framework and the network public opinion real-time analysis system based on Spark are designed and implemented,which focused on the real-time acquisition and analysis of network public opinion in colleges and universities,which provided an important basis for the analysis and research of network public opinion in colleges and universities.