Research And Application Of Big Data Analysis Based On Complex Network

Big data analysis is an important problem in big data research.Previous subject-oriented data analysis methods are preferable to mine intrinsic regularity of data for a given subject.However,big data is usually massive,heterogeneous,complex associated and open,and application-oriented big data analysis often involves multiple complex and dynamically associated subjects.Therefore,it is difficult to apply subject-oriented data analysis method to obtain the hidden systematic rules in big data.Big data is usually generated by real complex systems,and can be viewed as data image of complex system in information space.Data entities identify individuals in complex system,and relations between data entities can reflet interactions between individuals.It is an effective method to obtain systematic rules and mechanisms by studying big data analysis from the perspective of complex system theory and methods.Complex network is an effective model to study complex system,and it is necessary to systematically model and analyze big data based on complex network.On the basis of previous studies,this study presents two methods of constructing complex data networks using correlations and similarities in data,and proposes a vector space based complex network model for big data analysis.In this model,data networks composed by data entities and their relations are mapped into multidimensional vector space,in which each multidimensional vector identifies a data entity or a relation between two data entities.Furthermore,a complex network scaling operation and multidimensional analysis framework based on hierarchical transformations of multidimensional vectors for big data analysis is studied.Moreover,this study analyze correlations and similarities in water factor monitoring data collected from the Bohai Sea and the North Yellow Sea of China,and construct water factor correlation network and water factor similarity network of monitored areas.Further,the topological characteristics of these networks are respectively analyzed from spatial and temporal dimensions,and water quality of monitored areas are systematically assessed,and mechanisms of water eutrophication and red tide occurrences are revealed.Verifications and validations from actual released data show that the modeling and analysis method based on marine data network proposed in this study can effectively solve marine big data analysis problems,and contribute to water quality assessment,mechanisms discovery of red tide occurrences and other complex problems in marine field.Contrast with previous studies,this study propose a systematic framework for modeling and analyzing big data on the system level,which contributes to mining the hidden systematic rules and mechanisms in big data.