Design And Implementation Of Massive AIS Message Data Mining System Based On Cloud Computing And Distributed Technology

With the rapid development of maritime trade,AIS(Automatic Identification System)is becoming more and more popular.On the one hand,it retains a large number of ship navigation data for Maritime Bureau and related systems.On the other hand,it also puts forward a higher demand on how to efficiently process the massive navigation data.Cloud computing and distributed computing are being used more and more widely in all walks of life.Through the powerful computing resources of cloud computing and distributed computing,it can effectively solve the problem of insufficient computing power in the field of data mining.Therefore,in order to achieve big data analysis of ship trajectory,we take the AIS data as research object,design and implement a massive AIS data-mining platform,by combining cloud computing,distributed technology and data mining methods.The main work includes the following aspects:(1)Designed the technical solution of massive AIS data mining system on commercial cloud computing platform and distributed computing platform Hadoop.Developed subsystem to collection AIS data from different area safety and real time.Implementation high data concurrent writing and query subsystem use HBase.Designed data warehousing to interactive analysis of AIS data by OLAP.(2)In the cloud computing and distributed environment,in order to realize data mining and analysis of massive ship trajectory,we design and implement a parallel DBSCAN algorithm using distributed computing,and uses KD tree spatial partitioning algorithm to divide the data space.In order to realize the real time perception and abnormal alarm of the sea traffic flow potential,we design a real time cluster analysis method for ship monitoring area by using Spark Streaming.(3)The system is verified by the historical data of Zhoushan sea area and the real time AIS data of the Laotieshan.The experimental results show that,compared with single computing system,the system which used cloud computing and distributed technology can improve computing efficiency of vessels trajectory data mining.IT also shows that using Spark Streaming calculation to clustering vessels trajectory in real time can dynamic discover vessels' abnormal behavior.It further proves that our research has a high practical value in the field of Maritime Intelligent Regulation.