Research And Application Of Small Files Storage Method Beased On HDFS

With the development of science and technology, the traditional storage method has been unable to meet the current needs of massive data access, storage and processing of massive data has become a new topic of current research. Hadoop distributed computing platform because of its excellent characteristics of open source, high-throughput, high reliability and scalability, etc., have been widely applied in the field of cloud computing. HDFS distributed file system is one of the core technology of Hadoop, low requirement to the computer hardware, and has high fault tolerance, has been widely applied to store large data files. However, HDFS is designed for large-scale massive data files, in terms of access to a small file performance and efficient enough. In this paper, the problem of inefficient small HDFS file storage, combined with the system of public security and practical platform for cloud storage requirements, propose viable small file storage solutions to improve the overall system performance.Firstly, in order to solve the HDFS store lots of small files to the Name Node memory resources cost problem, after an analysis of the small file storage method for HDFS, design a separate small file processing module, the massive small files are merged before uploading to HDFS, so as to reduceing the burden on Name Node memory storage and improve storage efficiency.Secondly, in order to meet the high security requirements of public security system, put forward by the metadata for small file backup added in the clusters in a data server, to decrease when Name Node failure may occur when the information loss, improve the security of the system. In order to quickly access implementation after the merger of small files, proposing the creation of My Sql database in the data server for storing small files index information, providing access to an index of small files; and according to the correlation data in the public security system, proposed the small file metadata information prefetching scheme, improve reading speed of small files.Finally, through the simulation experiment, the optimized algorithm and HDFS algorithm with comparison of performance in both sides of small file storage and access, to verify the feasibility and effectiveness of the optimized algorithm.