Research And Development Of Food Safety Big Data Monitoring And Analysis System

In recent years,there have been a number of food safety incidents,which have aroused the attention and concern of the public and the country on food safety issues after the outbreak,and food safety has become the focus of people's attention.,on the other hand,with the rapid development of Internet in our country,the domestic Internet users has increased dramatically,weibo,WeChat,blogs and other media has become the main position of public opinion,and the relevant food safety information present a vast amounts of data scale,fast data transfer,a variety of data types,and value of low density of the four major characteristics,and in the acquisition,storage,management,analysis has been greatly exceeded the ability of traditional database software tools,so the food safety information monitoring analysis on prevention and control the occurrence and development of food safety events has important significance.This paper first introduces the research background and current situation of big data monitoring and analysis of food safety,and expounds the purpose and significance of designing a monitoring and analysis system of big data public opinion.Secondly,according to the characteristics and system requirements of public opinion on food safety,technical selection and module division are carried out.Then,in the implementation process,Scrapy based theme crawler program was used to collect public opinion data related to food safety on weibo,and a platform combining Hadoop and Spark for the storage and research of large amounts of data was built.This system adopts the method of deploying HDFS program on multiple local nodes to provide the ability to store a large amount of data,and analyzes and calculates the data through Spark,a fast and universal computing engine specially designed for large-scale data processing.Analysis including based on K-means topic discovery and Fasttext emotion classification,which due to the public opinion data mostly for short text,so in this paper,the existing essay this emotional tendency analysis algorithm is studied,found that as the information volume of swelling,the traditional method of judging emotional direction speed becomes slower,already cannot satisfy the user demand for high-speed processing a large number of essays in this document.Therefore,in order to solve this problem,this paper USES Fasttext,a word vector and text classification tool open source by Facebook,to judge the emotional polarity of public opinion data in the short text,and finds the parameters of Fasttext suitable for Chinese short text through experiments.This algorithm not only improves the accuracy of emotion classification,but also reduces the classification time from a few days to a few seconds.This paper developed and implemented a food safety monitoring and analysis system based on big data.This system can collect,store and analyze public opinion information of food safety,and display the development trend of public opinion and judge the polarity of public opinion in real time through the front-end page.At the end of the system development,the extensibility of the system is also fully considered,and space is reserved for adding other modules in the future.