Design And Implementation Of The Storage And Mining Subsystem For Urban Rail Transit Log Data

At present, Communication Based Train Control System (CBTC) is widely used in the urban rail transit. CBTC generates a large number of log data of electricity, locomotive etc., such as train number, locomotive state and fuel consumption parameters and so on. In this thesis, these data are called log data of the urban rail transit. With the rapid development and application of CBTC, it has become an urgent need for the effective storage and analysis of massive rail transit log data. And it has important practical significance for improving the transport capacity of urban rail transit, reducing operating costs and strengthening the safety operation and maintenance of urban rail transit.The big data technology is used to design and implement the storage and mining subsystem for log data of the urban rail transit, such as Hadoop, HBase and Spark. These big data technology has the advantages of high reliability, high scalability and distributed memory iterative calculation, etc.First of all, this thesis analyses the requirement of the whole storage and mining subsystem of urban rail transit log data and assigns the functional requirements of this subsystem, including data collection, data process, data storage and data analysis. Through the analysis of the above requirements, the thesis designs the overall structure of the subsystem and the division of function modules and then, introduces the interaction process between function modules in detail, and the abstract of the log data format and the main design of each module based on the characteristic of the log data. The core algorithm and code implementation of functional modules are given in the chapter of the detailed design and implementation. Then, the experimental test section describes the Hadoop and Spark cluster deployment environment and the test results of each functional module. Also, the system optimization scheme is presented in this section in view of the problem in the system implementation process. At last, this thesis summarizes the content and the author's work and a further improvement scheme is proposed.