The Hybrid Recommender System Based On Spark

With the rapid development of information technology,information overload has become an important challenge of Internet.In order to alleviate the growing contradiction between users and massive data,the researchers proposed the concept of the recommender system.As an important branch,a hybrid recommender system improves performance of the system by combining multiple recommendation algorithms.Currently the system is widely used in electronic commerce sites,social networks and video websites.However,a sharp increase in the amount of users and data has led to a higher requirements on accuracy and training time of a hybrid recommender system.For example,video websites require that a hybrid recommender system can recommend contents that users interested in accurately,and can also quickly update recommendations based on users’ behaviors and related information.Due to the increasing amount of data,it’s difficult for developers to determine the influence of each recommendation algorithm on hybrid results.Thus,a coarse-grained method for weight calculation is hard to meet users’ demand for accuracy.Moreover,a hybrid recommender system includes complex models and a large number of iterative computations,which significantly reduce the efficiency of the system.Thus,existing hybrid recommender systems can not satisfy users’ requirements for accuracy and efficiency.By analyzing characteristics of datasets,recommendation algorithms and method for weight calculation,we introduced a fast and general engine for large-scale data processing and implemented the hybrid recommender system based on Spark,aming at improving accuracy,diversity and efficiency of the hybrid system.The main work and innovations of the paper include:1.First,we proposed a fine-grained method for weight calculation.It calculates a weight vector for each recommendation algorithm.The method can not only improve accuracy of recommendation,but also alleviate the cold start problem caused by data sparse.2.Second,based on Spark and fine-grained weight calculation method,we designed and implemented the fine-grained weight hybrid subsystem.The subsystem reduces execution time based on the distributed computing framework Spark,and improves accuracy of the subsystem by using a fine-grained weight calculation method.Experimental results show that compare to a single recommendation algorithm,accuracy of the hybrid subsystem based on Spark increased by 5%-30%.Compare to the coarse-grained method for weight calculation,accuracy of the hybrid subsystem increased by 1.5%-3%.Moreover,training time of the hybrid subsystem is 90%lower than a stand-alone recommender system,and is nearly 2 times lower than the system which based on Hadoop.3.Third,we implemented a cross harmonic hybrid recommender system.The system introduced a content-based recommendation algorithm based on Spark.It is a high precision,high efficiency,high diversity and scalable hybrid recommender system.