Research And Implementation Of Hybrid Recommendation System Based On Spark

In the big data era,the recommender system has become an effective tool for solving the problem of information overload.On the one hand,the users screen information in the massive data and acquire effective decision support from the recommender system.On the other hand,the recommendation service providers,targeted personalized marketing by the recommendation system to increase revenue.In recent ten years,the recommendation system has made rapidly advanced development,but it still faces many challenges and problems,such as the storage calculation and scalability issues,the sparsity problem,and the recommended timeliness and so on.To solve the above problems,this paper studies and realizes a hybrid recommendation system for the film field based on the Spark platform.Firstly,the paper studies the matrix factorization method,and proposes a hybrid matrix decomposition recommendation algorithm which combines time factor and neighborhood information,and considers the issue that the interest of groups in changes as time going by.It also adopts momentum gradient descent method to solve the loss function,so that the convergence speed is improved and the prediction accuracy of the algorithm is increased at the same time.Secondly,an improved Pearson coefficient similarity calculation method is proposed to solve the similarity calculation problem of collaborative filtering,taking the influence of the popularity of the item and the individual bias into account.This method can effectively reduce the root-mean-square error of the algorithm.Thirdly,an incremental ALS algorithm is conducted for the timeliness of the recommendation system.For the newly acquired information,local modification of the model is taken to avoid the re-training of the model,saving a huge calculation spending.It is proofed by the experiment that the incremental ALS has a faster interaction speed and higher accuracy than the present popular incremental SGD,which effectively improves the response speed of the system.Finally,this paper designs and realizes a hybrid film recommendation system based on Spark,which includes the main modules of log collection,data processing,hybrid recommendation engine and etc.and combines the above optimization methods,effectively solving the prominent problems in the current recommendation system.