Research On Association Rules Algorithm For Massive Telecommunication Network Alarm Data Based On Spark

Telecommunication network alarm data is very important for network administrators to monitor and manage network equipment. We can get the network status by analyzing the alarm data, get the association rules which can help the network administrators by mining the network alarm data.But the alarm data becomes more and more.It has entered into "big data’’,the common data mining methods can’t process the big alarm data,we must find a new way to mine the alarm data by using some favorite big data processing tools.Based on China Mobile’s project which is about a telecommunication network analysis system, this paper proposes an improved assosication rules algorithm.To reach the point,this paper has completed the following work.1 .Reseached on big data processing tools such as Hadoop,Spark and so on.And this paper chooses the Spark after constrasting.Reseached on assosication rules algorithm , compared the classical algorithms Apriori and FP-Growth,and this paper chooses FP-Growth as the basic algorithm to modify.2.Accroding to the demand of China Mobile, this paper designed a telecommunication network analysis system which contains acquisition module, storage module, analysis module and display module.And I realized the storage module and analysis module.3. Analyzed the telecommunication network alarm data which provided by China Mobile.Based on the charactors of alarm data and Spark,this paper designed an improved assosication rules based on Spark which modifies the support count to filter high confidence and high lift telecommunication network alarm big data—FP-CUD.4.Build Hadoop and Spark cluster to test the FP-CUD and realize the alarm analysis system.The results show that the FP-CUD can mine out the strong association rules either in frequent patterns or non-frequent patterns, compared with the FP-Growth can only mine out strong association rules in frequent patterns.On the other hand,the FP-CUD based on Spark is more efficient than stand-alone and Hadoop.In addition,the alarm analysis system gives two statistical indicators which discard the "dirty data" or use the "dirty data" by using association rules,this provides the China Mobile more information to manage the network equipment.