Study On The Integration Of Multi Source Heterogeneous Marine Environment Data And Fishery Data

In 1980,the unfamiliar word,Big Data,was called the third wave of cadenza in the The Third Wave written by the famous American futurist Alvin Toffler.From that moment,this word,Big Data,began to appear on the world stage.From its appearance to becoming a hot topic,until the arrival of a new era,big data has penetrated into every industry of today’s technology Internet of Things,and now turns out to be a key factor in the development of science and technology.What followed was the mining and application of massive data.Our society is witnessing the rapid increasing of information,data is accumulating,and potentially useful information in data is becoming more and more abundant.It is difficult to extract information directly from the original data.Therefore,the effective data integration affect directly the efficiency of data mining knowledge’s extraction.With the continuous development of China’s offshore fisheries,there are more and more remote sensing data of fisheries obtained from satellites.How to use massive data to conduct corresponding research and analysis,and improve the level of fishery analysis in China’s fisheries has become one of the hotspots in researches of the national marine fishery sector and fishery companies.In the past research on the analysis of central fishery and fishery forecasting,the preprocessing of data usually uses excel,spss and other related processing software.However,in the context of big data,the data is diverse,simply using a fixed data pre-processing software is not suitable for all datas,so,it is very important to select appropriate preprocessing methods or software according to different data characteristics.In this paper,the South Pacific albacore tuna was taken as the research object,and the marine environment and yield data during the growth process were preprocessed.Then the environmental data and fishery production data are fused and superimposed to provide a data foundation for the later research on fishery forecast.In order to facilitatethe subsequent processing of data and achieve rapid data fusion,efficient retrieval,mutual conversion as well as effective matching,this paper established corresponding data specification standards,and designed a data integration management system.The main work of this paper is as follows:(1)Acquisition of marine environmental data and fishery operation data.The marine environmental data include: chlorophyll concentration,sea surface height,sea surface temperature,sea surface salinity,etc.Among them,the download format of sea surface height data is.NC format,chlorophyll concentration and sea surface temperature data acquisition format are.CSV format.The source of data is oceanwatch,the National Oceanic and Atmospheric Administration(NOAA)environmental database,and the Copernicus Marine Environment Monitoring Service(CMEMS).This paper writes a simple data crawler script,which can automatically download marine environment data.The fishery operation data is the South Pacific albacore tuna longline data in the form of.csv.Data were obtained from the website of the Western and Central Pacific Fisheries Commission(WCPFC)and Shenzhen Liancheng Oceanic Fisheries Group.(2)Due to the great amounts of data acquisition channels,the difference of data downloading formats and the force majeure factors such as occlusion by light or atmospheric clouds when acquiring data by satellite remote sensing,it is inevitable that there will be data loss and other problems under a large amount of data.For the problem of different data formats,this paper will convert the data format into.csv format.The goal is that the.csv data format has wider adaptability than the traditional.xls format..xls is a binary file,which can usually be opened only with excel.Whereas,.csv is a general file format.In the later stage,it can be easily imported into a database.In dealing with the data missing,the data characteristics of environmental data and fishery production data are analyzed,and three common geological interpolation algorithms are used to fill in the data missing.Through cross-validation analysis,the empirical Bayesian Kriging interpolation method is better than Pan-Kriging interpolation method and Local Polynomial interpolation method not only in environmental data interpolation,but also in fishery operation data interpolation.(3)For the pre-processed data,the three kinds of environmental data arenormalized by ArcGIS software,and the grayscale image with pixel value of 0-255 is generated.This paper,three kinds of environmental data are innovatively proposed as RGB three channel values,and the correlation between environmental factors and fishery CPUE is calculated by statistical and regression analysis methods,so as to determine the corresponding weight.On the basis of image fusion technology,each environmental factor is weighted accordingly and fused into a color map.Then the fusion map of fishery production and environment is superimposed to produce a map containing chlorophyll concentration,sea surface temperature,sea surface height and fishery CPUE data information,which improves the efficiency of data mining to extract effective information from images to a certain extent.The LeNet convolution neural network model is used to test the fusion graphs of two different correlation analysis methods.According to the analysis of the experimental results,it is concluded that the weighted fusion superposition graphs obtained by Pearson similarity in statistical methods are more suitable for research and analysis.(4)Designing data integration management system to achieve rapid data screening,query,the completion of data which is missed in the pre-process,and combining with the secondary development of ArcGIS,data filling and gray image generation realized,the final image fusion is carried out.With all these to provide users with efficient data storage,processing and access services.