| Complex networks have always been one of the classic fields of computer science research,scholars through the abstract of some concrete phenomena in real life as a complex network.Through the complex network topology,nodes or edges contain implicit or displayed information,to extract people interested and can bring great value to the community of information.For example,through the contact of people with complex networks,the sources of infectious diseases and the spreading of infectious diseases can be determined.Through the knowledge map,the trend of development of a certain subject and the cross-over with other disciplines can be obtained.In the field of complex networks,community discovery is the most basic and extensive research at present.Community discovery is based on the existing knowledge of the complex network is divided into several sub-modules,a module to become a community,which contains some close nodes and connecting edges.Community discovery can be understood as an unsupervised clustering algorithm based on complex networks.With the proposal of the concept of modularity and spectral clustering algorithm,community discovery algorithms developed rapidly in the early 21 st century,and various community discovery algorithms were proposed and argued.Outstanding community discovery algorithm can improve some indicator standards of the complexity of the network community division,from the performance can also improve the efficiency of discovery and reduce the time overhead.However,with the advent of the big data era,the scale of complex networks is also rapidly growing.In the actual operation,the computing power of single machine can not meet the needs of people,even if further optimization of the algorithm,single machine can not bring about a qualitative leap.As a result,the transfer of computing to big data computing platforms can satisfy the need for efficient computation of large,complex networks.Big data era has generated a number of big data computing framework platform.Among them,Spark because of computing speed,rich ecosystem,which won the big data industry widely recognized.Also,Spark Computing Platform has GraphX,a related graph computation component,that helps to synchronize algorithms.Based on the above knowledge,this paper proposes the research and improvement of community discovery algorithm based on spark for large-scale complex networks.This paper first weights a complex network with no weight.Then,this paper chooses the classic efficient community discovery algorithm-tag propagation algorithm to optimize the tagging,tag propagation and tagging strategies,iterative convergencestrategy and so on,and establishes a new community discovery algorithm model.Then,the algorithm is connected to Spark,the algorithm is synchronized through GraphX programming,and a Spark experiment platform is established.Finally,some classic complex network data and some large-scale complex network data are tested and compared with some classic community discovery algorithms to verify the proposed algorithm is validated and verified by a large-scale complex network data set based on Spark GraphX platform does greatly improve the computational performance of community discovery in complex networks. |
