The Research Of Real-time Data Cleaning In The Ship Monitoring System

With the increase of the number of shipping vessels in recent years, the safety requirements of the ship navigat ion are also improving. Real-time monitoring of the running state of ship's equipment is one of the main technical means to ensure the safety of the ship navigation. due to the large number and different categories of the devices, the harsh maritime navigat ion environment and the possible faulty of acquisition instrument, there will inevitably exist redundant and missing data in the whole monitoring dataset collected which may cause the wrong judgment of the sailing state of the ship. So this thesis focuses on the real-time data cleaning technology of ship monitor ing system, which mainly solves the problem of redundant and missing data in the monitoring dataset. The main research content of this paper is as follows:(1) The requirement analysis of the ship monitoring system is introduced, the framework of the system is designed and the key technology of the system is described.(2) An improved algorithm based on SNM algorithm is proposed to detect and elim inate the redundant data in the ship monitoring system. Based on the drawback of SNM which is that the size of sliding window is hard to select and the attribute matching is too frequent so the detection efficiency is unfavorable. The optimized algorithm is proposed By setting the size and speed of the sliding window variable to avoid missing record comparisons and reduce unnecessary ones, also it uses cosine similar ity algor ithm in attribute matching to improve precision of detection, and the Top-k effective weight filter ing algorithm is proposed to reduce the number of attribute matching and improve the detection efficiency. The experiment results show that the improved algorith m is better than SNM in recall rate, precision rate and execution time efficiency.(3) An improved algor ithm based on KNN algor ithm is proposed to impute the missing data in the ship monitoring system. The traditional KNN algorithm has a huge time consumption when computing the distance between missing data and the complete data in the ent ire data set, so the improved weight ing method based on reciprocal of multiple correlation coefficient is proposed to simplify the data set by elim inat ing some attributes unrelated to the missing attribute to reduce the time complexity. Also the calculation method of Euclidean distance in KNN algor ithm is too limited in attribute dimens ion, the improved algorithm combines Mahalanobis distance and Grey correlation analys is to select k-nearest neighbors, which has a good efficiency in both continuous and discrete data. Finally the concept of entropy weight theory is introduced to calculate the estimated value by computing the entropy weight of every relevant attribute, which increases the objectivity and accuracy of filling values. The experimental results show that the improved algorithm has a higher filling accuracy and stability compared with the traditional KNN algor ithm.(4) The project background of the ship monitoring system is introduced, the realizat ion of the key technology and application of the main functions of the system are given. The application shows that the research results of this paper are effective.