Research And Implementation Of Ocean Element Data Service Platform Based On Multi-source And Heterogeneous

In recent years,with the continuous development of computer science and big data technology,human society has gradually entered a new era of rapid development of data science.Under the background of this era,ocean data science based on big data technology can realize the integration and reuse of existing resources,improve the operating efficiency of the industry,and tap the huge potential of the industry.However,the multi-source and heterogeneous nature of marine environmental element data makes it difficult to achieve effective organization and management.Traditional marine element data sharing methods also have problems such as low visibility and single user interaction,which cannot fully utilize the economic and economic aspects of the data.Social value has greatly affected the development of ocean data-related industries such as deep-sea fisheries.Based on the characteristics of heterogeneous ocean element data,this paper proposes an ETL(Extract-Transform-Load)-based ocean element data integration scheme based on the existing data integration technology,and analyzes the abnormal data detection algorithm in the preprocessing process.And improvement,realized the ocean element data visualization service based on Web GIS technology.The main work of this paper is as follows:(1)A detailed analysis was made on the heterogeneous data sources of marine environmental elements,a data standard with strong compatibility was designed,and a multi-source heterogeneous marine environmental element data integration service program was proposed.Based on the requirements of data integration,the data integration module of heterogeneous ocean elements is realized,and data collection,conversion,and reporting tasks can be completed through a unified data access interface.In addition,real-time monitoring and management of data integration tasks,basic metadata,mapping metadata,and ETL metadata are realized through the web interface of the background management module.The module realizes the unified format conversion between different data sources and target data through the mapping of source data structure and data standard structure.Finally,an image resampling algorithm is used to verify the result of ocean feature data integration.(2)The traditional anomaly data detection algorithm K-means has defects such as insensitivity to global outliers and susceptibility to initial clustering centers,and cannot adapt to the characteristics of the increasing amount of ocean element data.In order to improve the efficiency of the algorithm for detecting abnormal points in ocean element data,the wolf pack algorithm is first improved by using the adaptive step factor and the taboo table,and then the optimized initial clustering center is selected by the improved wolf pack algorithm,thereby improving The performance of the K-means algorithm.Experiments have proved that the improved algorithm can better process the ocean element data set,and the comprehensiveness of outlier detection has been greatly improved.(3)Through the analysis of the actual needs of the system,combined with the research of the basic structure of the data service platform,we designed and realized a multi-source heterogeneous ocean element data service platform based on JAVA language and Web GIS,Redis,Spring Boot and other technology development.Including ocean element data integration module,ocean data rear management module(backstage management module),ocean element display module.It has realized the integration of multi-source heterogeneous marine element data,data integration task management,and the visualization of marine environmental element data,meeting the actual business needs of different users.