Design And Implementation Of Subway Operation Log Collection And Preprocessing System Based On Hadoop

When the subway system is running,subway signaling equipment will continue to exchange information with each other and generate a large amount of operation logs.Collecting and analyzing these logs can help the management of the subway system.At present,in order to identify the potential problems in subway system,maintainers check the operation logs manually.With the growth of subway mileage,the operation pressures will increase.In order to ease the operation pressures,it is need to adopt modern technology to collect and to analyze subway operation logs.This paper proposes a subway operation log collection and preprocessing system based on Hadoop.The system can collect operation logs of different subsystems from different subway lines,and provide data support for subsequent data analysis by preprocessing the logs.The paper analyzes the characteristics of subway operation logs and determines the system's design goals.The design of entire system is divided into three parts:data collection module,data preprocessing module and system reliability design module.Data collection module used FTP and Flume to collect the operation logs,and then store the logs in Hadoop cluster.Data preprocessing module consists of two parts:data splitting sub-module and data integrity check sub-module.The function of Data splitting sub-module is mainly to split operation logs in the cluster,and data integrity check sub-module can ensure the accuracy of the logs.System reliability design module includes the reliability design of FTP and Flume,it can make the system with the ability of fault tolerance.At last,the paper proves that the system can collect and preprocess operation logs of different subsystems from different subway lines through tests and analysis.