Research On Key Algorithms And Technologies Of High Performance Distributed Storage System

With the rapid development of the Internet era and the advent of the era of cloud architecture,a large amount of data will be generated at every moment.How to achieve a high-performance and high-efficiency distributed storage system at a lower cost is a challenge for the storage industry.This thesis focuses on performance issues in the field of distributed storage,explores the optimization space of key modules and algorithms in the storage process,and proposes feasible performance optimization strategies and techniques.In order to meet the current demand for classified management of hot and cold data,and to reduce storage costs as much as possible under the requirements of high data throughput performance,by studying and comparing existing distributed storage systems,this thesis proposes a distributed storage system developed by the Erlang/OTP framework,and key modules is researched deeply.The system in this thesis implements classified management of cold and hot data through three functional nodes,and achieves data consistency through distributed consistency algorithms and erasure coding strategies,ensuring that the system can provide normal storage service when some links fail.In terms of performance,the system has made corresponding optimizations in read-write caching,file version control,and load balancing strategies,and continuously pursues higher storage performance on the basis of data security and integrity.This thesis focuses on the research of distributed consensus algorithms in the field of distributed storage.By comparing traditional distributed consensus protocols,the Raft protocol is studied in depth by combining theory and practice,and the Multiple Raft storage consensus algorithm is proposed on this basis.In terms of system scalability,availability,cluster rebalancing,etc.,optimization strategies have been made for the storage field.Experimental tests have verified its excellent cluster availability and scalability,and data storage performance has been significantly improved.In order to ensure the flexible deployment and integrity of the storage system,this thesis uses the Golang language to design and implement the system storage client.The client uses the FUSE file system as a development framework,combined with the consistent hash algorithm to maintain business load balance,and explains the operating mechanism and technical processing flow of a series of processes ranging from the operating system user space sending read and write requests,to the client interface processing and network topology,to the data placement of the back-end storage service cluster.