Design Of Bus IOT Mass Data Storage Platform Based On Hadoop

With the continuous development of public transport and networking technology,the amount of data of various types of vehicle information collection device generated gradually increase, traditional data storage methods have been difficult to support massive log data and distributed storage bus surveillance video, so how these tools storage and manage data efficiently is an urgent problem. From the bus was collecting data networking features are view, GPS log data used for analysis to optimize bus scheduling and bus lines, so not only need to implement a distributed log data compression and storage but also need be supported to a range of time and space efficient data logging scan; buses IOT surveillance video has complex sources,confusing format features, in order to achieve unified management of video surveillance data storage, we need to implement a distributed video file transcoding.To achieve the above requirement, respectively, according to the characteristics of the bus Things GPS logs and video data is designed based on Hadoop distributed file storage system, an effective solution to efficiently store and manage data bus networking. Firstly, this paper based on the existing technology analyzes the Hadoop architecture, combined with the Public transport network system Hadoop platform and gives the background of public transport network huge amounts of data distributed storage system solutions.This paper studies the characteristics of the technical architecture of Hadoop, and analyzed and summarized the public collection system of things GPS and video data logging, processing and storage characteristics, based on Hadoop distributed storage networking solutions bus massive data collection.The daily vehicle operating conditions were analyzed, and the demand for GPS log list when using the mainstream of data storage solution before the advantages and disadvantages, and design a set of GPS based on Hadoop log data storage management system. The system according to the time and GPS location information will log data hierarchical classification, storage and use of compression techniques in HDFS. The system also presents a according to the methods of dynamic data storage nodes were performance, can effectively improve the efficiency of log file writing and storage, meet the use of Public transport network system requirements. This method through the experiment, confirmed the validity of the method and the stability of the system.Based on-board monitoring video storage space and video formats complex problems, we design a Hadoop distributed video transcoding storage system. Video storage system uses HDFS, use of graphs and FFMPEG to realize distributed transcoding,;utilizing LZO compression technology to achieve high compression of video files stored. In the end, through the experimental results show that the system can greatly improve the efficiency of transcoding.