Design And Implementation Of User Log Collection And Real-time Monitoring System For Big Data Platform

With the rapid development of mobile devices and applications,various Internet service big data platforms have prospered and developed.Massive users on big data platforms will generate large amounts of log data.How to collect and real-time monitor the massive user log data scattered on the system side and the terminals,and pay attention to the important indicators in the log in real time has become a problem that big data platform companies must solve.Through the monitoring of log data,people can instantly understand the equipment system of the enterprise big data platform,detect and troubleshoot possible failures,optimize business processes,and provide better services for platform users.At present,the big data platform faces the following major problems in user log processing:(1)A system is needed to be more in line with the actual business needs of the enterprise,such as high throughput,reliable data transmission and other characteristics,to complete the collection,cleaning and cleaning storage of logs.(2)Access management of real-time log data is inconvenient,and it is difficult to easily and effectively manage log historical data,which affects the efficiency of upper-layer data query and statistical analysis calculation.(3)Joint debugging of current general real-time monitoring and alarm business development take a long cycle,and a system tool is needed to undertake flexible and efficient log data monitoring tasksThis article first explains the significance and related applications of the log collection and real-time monitoring system for big data platform companies in the current big data environment,and then explores the necessary characteristics of the system architecture for the distributed logs from the needs of big data platform companies for user log data collection and effective monitoring;then design the system's functional architecture,technical architecture and network architecture,and the technical selection of the development tools used in the system.Finally,the functions of each module of the system are elaborated in detail,and the log collection and real-time monitoring system is realized in combination with the system architecture and detailed design.The system can be divided into the background log data collection,cleaning and storage part and the foreground user log management and monitoring part.The foreground part can be divided into the basic connection management module,the Druid data source configuration module for log data access and data management,and the monitoring task configuration module for monitoring tasks and monitoring script creation.The log mentioned in this article is a customized log defined by the application system through the log strategy.The background data processing component can complete the overall process of log data collection,cleaning and storage.At the same time,the developer can send the log data from the Kafka message system through the foreground web side of the system to the real-time computing engine Druid and manage it for subsequent monitoring tasks or real-time query.For the log data that is accessed,log monitoring tasks can be created,and monitoring scripts can be generated and deployed to the Airflow workflow platform to complete the real-time log data periodic query and alarm work.This paper uses the online test environment to test log collection and real-time monitoring system and achieves the expected design goals.This system has been launched on the big data platform of a large domestic Internet company.The system has well realized the management of user log data sources in a distributed environment and the effective configuration of real-time monitoring tasks for user logs,which can easily meet the needs of enterprise developers and business requirements for user log data management and real-time monitoring.