| With the rapid development of science and technology,we are enjoying much convenience in all aspects of life.The global electronization has made e-mail technology useful.E-mail has got rid of the traditional paper writing.What's more,it can transmit information more quickly.The cost of manpower and material resources has also been declined.In addition,it is very easy to retain and hard to lost.For all above reasons,e-mail has become one of the most popular means of communication in the current era.While e-mail has brought about many conveniences,it also leads to many drawbacks.A large number of spam began to hinder our daily life.E-mail is being used by illegal vendors to spread viruses,pornography,rumors and other information,which caused great inconvenience to users' lives and jobs.Meanwhile,it damages the security of the network.Furthermore,spam can also bring about communication congestion and other issues.For example,a large number of emails restrict the memory space and computing power of users' computer,which reduced the processing speed of a computer.Under the background of big data,the dimension of data volume and feature space increases rapidly,and the parallelization of text categorization algorithm tremendously improves its efficiency.In this dissertation,a bayesian spam classification system based on parallel computing with Spark is proposed.It combines Simhash de-duplication algorithm with naive Bayesian classifier,as well as using Resillient Distributed Dataset(RDD)model to process e-mails in parallel.On the one hand,it enhance the classification effect of the system.On the other hand,it improves the capacity to deal with mass e-mails of the system.The main research works of this dissertation are as follows:1.Analyse and compare the commonly used spam classification algorithms.Among them,Naive Bayesian classification algorithm is widely used because of its advantages such as fast operation,simple algorithm and accurate classification.Thus,Naive Bayesian classifier is determined as the core part of spam filtering.2.Using RDD programming model to realize the parallel processing of computing process during the whole training and classification stage of Naive Bayesian classifier,which effectively improves the running speed of mail classification system.3.A single classification method can only cover a certain type of spam.With the increase of the number of e-mail,these processing technologies are obviously inadequate in performance so that Simhash de-duplication algorithm is applied to the system.Before e-mails enter the core classifier module,Simhash algorithm is used to filter them at first,and then uncertain mails enter the naive Bayesian classifier to be determined.In this way,the classification accuracy of spam classification system is improved and the processing time of the system is shortened.4.By comparing the experiments of spam classification system,in Simhash and Naive Bayesian classifier,both of the recall rate and accuracy rate of the system are close to 94%,which is much higher than that of the system which only use Naive Bayesian classifier as filtering module.At the same time,compared with the single-machine mail classification system,the parallel computing of Naive Bayesian with Spark technology also greatly improves the speed of mail classification system. |
