Desigh And Implementation Of Water Traffic Data Analysis System In Big Data Environment

The data sources of water traffic are complex and diverse,and the data structure is complex as well as the volume is large.Therefore,the data processing process is relatively complex,and the efficiency and automation degree are relatively low.This paper mainly studies how to improve the processing performance of water traffic data,and in the complex and diverse data structure can be flexible,multidimensional,accurate data analysis and statistics,and use visual form to show the results of data analysis and statistics,to help water traffic information researchers make better decisions.Water traffic data sources are mainly divided into AIS data collected by equipment and business data reported by various water traffic units.Therefore,different processing schemes are developed in this paper for these two types of data sources,and unified analysis and display are conducted in the system after processing respectively.Research work mainly include the following:1.In view of the equipment of automatic acquisition of AIS structured real-time data analysis:after using the Scala language to parse the data preprocessing,the use of relational database document Hbase for data storage,and focuses on the section dimension of ship traffic statistics algorithm,using the algorithm combining the Hadoop distributed system based on pretreatment of AIS data calculation engine spark calculation,finally the calculated results stored in the relational database,and in the system to realize the map in the form of visual display.2.Analysis of relational business data:as the data structures reported by different traffic units are greatly different,complex and changeable,and joint statistical analysis is finally needed,relational database SqlServer is used to store business data.In this paper,metadata engine is used as the data access point to drive the data as a whole.At the same time,the multi-dimensional analysis of the massive historical data has achieved the comprehensive,rapid and visual analysis and statistics requirements of the business data.3.On the basis of the above two data processing schemes and the analysis of the requirements obtained from the survey,the modules of data management,data processing,rule management and data analysis and statistics were designed,and the data processing and analysis system based on B/S architecture was finally realized.Therefore,the water traffic data analysis system under the big data environment studied in this paper can obtain traffic information more efficiently and accurately,which meets the requirements of water traffic information researchers for comprehensive,fast and multi-dimensional data analysis.