| In the wave of big data brought by the Internet,the correlation between people and the network has been greatly deepened.People have become accustomed to socializing,learning and understanding the world through the Internet.Different from the "read-only" nature of traditional media,many online media information has a comment function,leaving public opinions and views on a matter,and individuals can also share it.own creation and content,so the Internet has become a super-large information gathering platform.But the distribution of information on the web is messy,and there is a lot of fake news that misleads people and makes it hard to see what an event really is.Therefore,a public opinion system that can monitor the information on the network platform in real time and give intuitive results through calculation and analysis is very necessary.The public opinion system came into being with the development of the network.The purpose is to replace the manual and automatic collect,process and analysis network data with machines,which is far more efficient than manual processing.With the development of the public opinion system,the amount of data monitored has become larger and larger,and its functions have become more and more abundant,and many special functions have been derived for different needs.The current public opinion system is mainly developed by large companies,which is expensive to use and complex.This paper is mainly oriented to the public opinion system for monitoring hot events,and mainly involves the following technologies:data collection,text clustering,sentiment analysis and visualization of public opinion results.Among them,the most used solution for data collection is to use crawler for data crawling;text clustering groups the texts belonging to the same event;sentiment analysis is used to obtain the emotional tendency of the event;result visualization can intuitively display the results of public opinion for People directly view real-time public opinion.At first,text clustering technology mainly aimed at non-incremental text data sets.All texts in the set were uniformly read and then iteratively calculated and divided into categories.However,this obviously did not meet the needs of real-time data reading in the public opinion system.The incremental clustering algorithm does not look back on the historical text to optimize the clustering results,so the clustering accuracy is not high when clustering a large amount of data text,and the previous errors will always affect the subsequent clustering process.Sentiment analysis is one of the popular directions in the field of natural language processing.The emergence of BERT has also promoted the progress of sentiment analysis tasks.However,the BERT model is too large and will occupy a lot of overhead in practical applications.It is not practical for some small and medium tasks or systems.This paper designs and implements a distributed public opinion detection system for hot events according to the needs and existing problems of the public opinion system.The main work is to use the Scrapy crawler framework to implement an automatic and scalable crawler based on the Weibo platform,which solves the information source problem of the public opinion system and facilitates expansion;Design the SinglePass&K-Means combined clustering algorithm to solve the incremental increase of the public opinion system data Compared with the single-passSOM clustering method,the accuracy of the clustering method is improved by 6%on average;the sentiment analysis model is obtained by training the DistillBERT pre-training model on the Weibo comment data set,avoiding the direct use of the model in the system.The cost of BERT is too large.While the accuracy rate is only reduced by 0.5%,the size of the model is reduced by half,and the training and prediction time is also shortened by nearly 50%;a large-screen display interface is built using the Flask framework and front-end technology,which is intuitive It displays public opinion results such as recent hot events,event content,event development timeline,and event emotional tendencies;uses distributed services to deploy each module separately,makes full use of computer resources,and improves the overall performance of the system. |
