A Real-time Recommendation System Based On Flink

In the field of education,the degree of integration of various online learning websites and APPs with recommendation algorithms and systems is still relatively lacking.In my country’s teaching industry,exercises are an important and indispensable teaching resource.In the vast sea of questions,there are many questions,filtering out redundant information that learners do not need,and recommending them according to learners’ hobbies and ability levels Appropriate exercises for training are very important.Maintaining a good state of learning can keep people in the "zone of proximal development" of learning,stimulate learners’learning ability and enthusiasm for learning,and is of great significance to promote their learning progress and the vigorous development of the education industry.In this thesis,after fully investigating the current situation of the field of online learning and personalized recommendation at home and abroad,combined with certain pedagogical theories,the learning efficiency of learners is increased by combining exercises and recommendation systems.First,this paper formats the exercise data source and the collected user data,extracts the eigenvalues required by the algorithm,and converts the eigenvalues into a matrix,and calculates the user’s expectation of the exercise in the eigenmatrix.Finally,the offline recommendation algorithm of the system will be implemented using the collaborative filtering algorithm.The underlying computing engine chooses to use the current mainstream big data computing engine Apache Spark.Its micro-batch processing method,data partitioning and automatic partitioning functions can effectively Reduce and reduce the data transmission pressure between the working nodes in the big data cluster and the calculation pressure caused by the Cartesian product when the data is converged.It loads the data into the Spark memory for operation,which can achieve fast data processing and timely response.Task requirements can improve work efficiency.In the selection of the real-time recommendation algorithm,due to the long computing time and low efficiency of the collaborative filtering algorithm itself,it cannot achieve fast feedback to users when used for real-time recommendation.The IDF algorithm calculates the similarity of each exercise in the data set.Considering that the user’s preferences will shift with time,it is decided to introduce a time weight factor into the user’s interest.The user’s interest formula thus created is not only It can satisfy the recommendation of the user’s favorite items and ensure the real-time performance of the system.On the underlying computing engine,it is recommended to use the streaming computing platform Apache Flink in real time,which supports high throughput,low latency,and high performance.It also has state computing capabilities and rich time window semantics,which allows us to use flexible trigger conditions to customize Streaming of complex data.Finally,based on the above education theory and recommendation algorithm theory,this thesis selects the current mainstream big data architecture Kappa architecture and matches related components,including the web front-end framework Vue,the back-end business framework Spring,log collection tool Flume,message queue Kafka,data Storage systems Mysql,ES,data lake Iceberg,etc,finally developed a set of offline and real-time exercise recommendation system.