Recommendation System Based On Spark And Hybrid Weight Algorithm

The development of Internet technology is changing with each passing day.People used to worry about lack of information.Now they face huge amounts of information but it is difficult to get useful information.People's worry has changed from the lack of information to how to obtain the valuable information they need in the sea of information.The emergence of the recommendation system solves the problem of information overload to some extent.From the initial single algorithm recommendation system to the later traditional weight mixing system,the recommendation system has developed very rapidly,and has mature applications in the fields of e-commerce and news push.The traditional weight hybrid algorithm system often has better performance than the single algorithm recommendation system.The weight parameters of traditional hybrid recommendation systems are usually obtained by researchers' experience or through a lot of experiments.With the increase of data,the requirements of parameter accuracy and efficiency are gradually improved,and the previous parameter setting methods are gradually unable to meet accuracy and efficiency.Secondly,in the face of massive data,traditional data distribution platforms often have the disadvantage of too long training time due to a large amount of I/O overhead.And the traditional backup fault-tolerant design wastes a lot of memory.Finally,the problems of data sparsity,cold start of the system and unsatisfactory prediction accuracy still exist.Based on the inconsistent bias of different recommendation algorithms and the introduction of Spark,a new generation of distributed big data computing platform,this paper implements a new hybrid algorithm recommendation system to achieve more efficient and accurate recommendation.The main work and innovation points of paper are as follows:(1)This paper designs and implements a new weight calculation method,and the weight is assigned according to the bias of each algorithm,which can well alleviate the accuracy of weight assignment in the traditional mixed system and improve the scientificity and accuracy of the weight.(2)In the paper,several single recommendation algorithms are designed and implemented on the distributed computing platform Spark.Combining single algorithm and new weight calculation method,a hybrid recommendation system is implemented.The experiment shows that the hybrid algorithm recommendation system based on Spark has significantly improved the recommendation efficiency and accuracy compared with the traditional platform and algorithm.(3)This paper designs a new recommendation system.The system includes data input output,algorithm calculation and recommendation.Based on the high efficiency of the platform itself and its RDD pedigree mechanism,this paper implements a more efficient,more accurate and more secure recommendation system.