Small Files-oriented Distributed File Storage Management System

In the 21st century,with the continuous improvement of the mobile communication network environment and the rapid popularization of smartphones,the mobile Internet has rapidly risen.The mobile Internet generates massive amounts of small files such as texts,pictures,audio and video every day.Currently,distributed file storage systems represented by GFS and HDFS are mainly designed for storing massively large file,and lack the targeted optimization of small file storage management.Therefore,existing distributed file storage systems have problems such as limited capacity of metadata servers,inefficient system access,and low storage resource utilization when storing small files.In addition,the periodic dynamic feedback load balancing algorithm mostly used in the current distributed file system has some shortcomings.In this algorithm,the storage node periodically feeds back the load information to the scheduling node,but the length of the feedback period is difficult to determine.If the feedback period is too short,extra network overhead will be caused and the network load will be increased.If the feedback period is too long,the real-time performance of scheduling nodes for obtaining load information will be reduced.To solve these problems,this paper proposes a new load balancing algorithm for distributed file systems,and designes a distributed file storage management system for small files.In the design of this system,the underlying storage module is implemented based on FastDFS,and combined with Elasticsearch to achieve the multi-dimensional search of files..The main work of this paper includes:(1)In order to solve the existing problems of load balancing algorithm in distributed file system,a new adaptive dynamic feedback load balancing algorithm based on QoS is proposed,named QoS-ADFLB algorithm.(2)This system uses FastDFS as the underlying storage service,encapsulates the native interface,realizes the storage of large amount of small files and concurrent access of multiple threads.(3)Develop corresponding attribute extraction strategy for different types of documents,to generate the document attribute sets.(4)Use Elasticsearch to index and store document attributes sets,achieving multi-dimensional and fast retrieval of documents.(5)Program to implement the distributed file storage management system for small files.In order to verify the feasibility of this project,the environment configuration and system deployment are completed on the server cluster.The function and performance of the small file-oriented distributed file storage management system which is designed and implemented by this paper are verified and tested.The experimental results show that the system can realize the efficient storage and quick access of the large number of small files,and support the extraction,indexing and retrieval of the attribute sets of different types of files.At the same time,the actual distributed file system environment is set up on the server cluster.The performance of QoS-ADFLB algorithm is tested through the comparative experiments.The experimental results show that QoS-ADFLB algorithm can effectively reduce the average response time of the system and improve throughput,make system load to achieve a good balance.