Research And Implementation Of Recommendation Algorithm Based On Spark

With the continuous development of the Internet in the 21 st century,more and more information is available,and the problem of information overload is becoming more and more serious.It’s getting harder and harder for users to find things that appeal to them.The recommendation system is a good solution to the problem of too much information.The recommendation algorithm of Latent Factor Model and the collaborative filtering algorithm based on items are proposed and applied to the recommendation system,which can effectively improve the accuracy of the traditional recommendation system.However,the traditional Latent Factor Model has some problems of cold start and sparse data.For new users,the recommendation system cannot generate recommendations.Secondly,the user score matrix data is missing,leading to low accuracy of prediction.The collaborative filtering algorithm based on items consumes a long time in real-time recommendation and cannot be applied in real-time recommendation.At the same time,these two algorithms do not take into account the influence of time factor on users’ interest.To solve the above problems,this paper improved and implemented the recommendation algorithm based on Spark platform,and the main research work of this paper is reflected in the following aspects:Firstly,in order to solve the problems of cold start and sparse data of traditional Latent Factor Model,this paper proposes a Latent Factor Model incorporating user information and time factors.In this model,the information provided by new users when registering is integrated into the Latent Factor Model,and the user score matrix is modified by time function before solving the Latent Factor Model,so that more accurate recommendation can be made.The Experiment was then performed on the Spark platform using the Movie Lens dataset.Experiments show that the improved Latent Factor Model can achieve more accurate recommendation.Secondly,to solve the problem that the traditional collaborative filtering algorithm based on items takes a long time to calculate when it is applied to real-time recommendation,this paper improves the traditional collaborative filtering algorithm based on items by decomposing the similarity matrix of items,integrating the users’ score of items,and introducing the reward and punishment functions and time weights to calculate the recommendation priority of items.It improves the response speed of the real-time recommendation to a certain extent and also has certain accuracy.Finally,this article is based on the above two kinds of improved algorithms,with improved Latent Factor Model as offline recommendation algorithm,with improved collaborative filtering algorithm based on item as real-time recommendation algorithm,designs and realizes a movie recommendation system,based on the Spark for the system function test,system response is accurate,and recommend the effect is good.