Design And Implementation Of Movie Recommendation System Based On Spark And User Preference

With the rapid development of the society,the scale of the internet is expanding,and the network data is also growing explosively.Increasing data will inevitably increase the difficulty of data acquisition,and information overload has become an urgent problem to be solved on the Internet.In order to help people get data quickly and accurately,recommendation system has become an important research topic in the field of computer.Recommendation algorithm is the core of recommendation system.Based on the research of recommendation algorithm,a collaborative filtering recommendation algorithm based on clustering and user preference(CP-UBCF)is proposed to solve the problems of sparse rating data and scalability of the collaborative filtering recommendation algorithm.CP-UBCF designs a similarity calculation method based on user preference.This method not only considers the user’s own attributes but introduces the user’s rating and the occupation’s rating of the item type.At the same time,in order to reduce the sparsity of user-item rating data,CP-UBCF designs a Weighted Slope One algorithm which integrates user preferences to predict and fill in unrated items of user-item rating data.In addition,in order to improve the scalability of the algorithm,CP-UBCF designs a Den-Dis K-means clustering method when processing rating data.The Den-Dis K-means clustering method is designed based on density and distance in order to solve the clustering instability problem caused by the random selection of initial cluster centers of K-means clustering.By this clustering method,clustering users can reduces the search space for finding similar users during the recommendation process.Finally,the user-based collaborative filtering recommendation algorithm is used in the clustered data set to generate the recommendation results of the target users.According to the experimental results of Movielens 100 K movie rating dataset,the recommendation effect of CP-UBCF algorithm has been improved.Based on the proposed recommendation algorithm,this paper designs and implements a movie recommendation system based on Spark,Elasticsearch and CP-UBCF.The recommendation system effectively uses Spark’s big data computing performance and Elasticsearch’s indexing performance while ensuring the recommendation effect through CP-UBCF,which meets the needs of actual application scenarios.