Research And Implement Of FastDFS-based Efficient Access Method For Massive Small Files

Distributed file system provides an efficient solution for large file access.With the rapid development of the mobile Internet and big data,distributed file systems have run into issues such as low throughput and high latency when utilized for massive small file access.It is critical to develop innovative ways for distributed file systems in order to increase the performance of small file access.Starting with the characteristics of small file batch access with attribute association,this dissertation investigates the small file access process and load balancing algorithm,proposes the MAB strategy for small file access and a dynamic load balancing algorithm based on residual performance which are applied to FastDFS,and finally designs and implements a file management system for small files.A list of the specific work is provided below.(1)To address the problems of random IO caused by decentralized storage of small files and additional disk overhead caused by metadata access,this dissertation proposes the MAB method based on FastDFS’s small file merging function,which includes three steps of same attribute small file merging,asynchronous write module,and file batch processing.FastDFS has excellent results after applying the MAB method,with an average performance improvement of 4.8 and 4.2 times over the original and an average performance improvement of 3.7 and 3.4 times over the SCP remote file transfer command for small file reads and writes.(2)This dissertation proposes a dynamic load balancing algorithm based on residual performance and applies it to the two stages of Storage selection and Group selection of FastDFS to address the problem of uneven node load and storage space allocation during the allocation of IO tasks by static load balancing algorithms.The algorithm considers a variety of criteria that affect the nodes’ ability to handle IO requests,as well as available storage space.In the best-case scenario,the algorithm reduces response time by 29% compared to the polling method and 17% compared to the weighted polling algorithm,and storage space distribution between Groups is more acceptable.Finally,this dissertation proposes a file management system for small file storage based on FastDFS and the aforesaid enhancement method,which uses My Sql to manage directories and file metadata and FastDFS to store small files.From the standpoint of software engineering,this dissertation introduces requirement analysis,overall design,and primary module design,as well as elaborates on system deployment and exhibits the system prototype.Using Open Data’s license plate picture dataset,the system is tested for file access performance.Under the same hardware settings,the experiments show that the system can support larger concurrency and has lower response latency.