Research And Implementation Of Rainfall Prediction Method Based On Spark

The satellite cloud image can directly reflect the motion state of various cloudsystems,and it has become an important reference for climate and hydrological prediction.Over the years,the water industry has accumulated a large number of satellite cloud images and rainfall data.The method of manual interpretation for it is not only inefficient but alse qualitative,this way has low credibility in the decision-making process.In order to solve this problem,this paper comprehensively applies Spark technology and image mining technology,and analyzes the relationship between cloud images and rainfall by association rules analysis,which provides theoretical basis for rainfall prediction based on satellite cloud image.The main contents for research and development of the thesis as follows:(1)Constructed a relationship model between cloud image data and rainfall data.Based on grayscale features designing five kinds of cloud image feature parameters,which is related with rainfall,then,builded multi-dimensional cubes for cloud image feature parameters and rainfall data,and used a clustering technology to divide mixed data into different regions,then,exploring the relationship between cloud image features and rainfall data by using association rule mining algorithm.(2)Using clustering technology to separation and extraction of cloud information in satellite cloud image.studied the Canopy-FCM algorithm and introduced the coarse clustering Canopy algorithm is to improve the fuzzy C-means algorithm.then,for the problem of local optimum,combined with the principle of maximum and minmum,and the MMCanopy-FCM algorithm is proposed.In order to support the processing of massive satellite cloud image,this paper proposed an d implemented the parallelization algorithm SP_MMCanopy_FCM algorithm,what is combined with the Spark programming model.The experiment proves that the algorithm is effective for cloud layer separation.(3)Based on cloud map-rainfall mixed data set mining something what is association rule.Combining the matrix characteristics to improved the Apriori algorthm,and the MC Apriori algorithm is proposed base on Matrix compression,it overcome the shortcomings MC_Apriori algorithm is proposed base on Matrix compression,it overcome the shortcomings of generating too many useless candidate sets and improve utilization of memory.In order to meeting the need for mining association rule between massive cloud image and rain data,The Apriori algorithm is optimized in parallel based on the Spark platform,and the SP_MC_Apriori algorithm is proposed.In experimental analysis using the mixed dataset of actual cloud image and rainfall,it shows that the method proposed has good accuracy and time efficiency in this paper.(4)Based on the above methods,designed and implemented the prototype system for rainfall prediction based on Spark big data platform.