Design And Implementation Of Distributed Storage System Based On Raft Protocol

The advent of the mobile Internet era has brought about rapid development in internet technology,leading to the emergence of large-scale applications represented by big data and cloud computing.These applications have higher requirements for data storage capacity,efficiency,and security.Compared with the traditional centralized server storage,distributed storage has significant advantages in storage efficiency,access speed,security,and storage capacity.However,it introduces complex issues related to data consistency.Paxos and Raft are the two most widely used algorithms for solving data consistency issues in the distributed field.The Raft not only improves understandability,but also exhibits comparable performance,reliability,and availability to Paxos.Due to the strong leader property of the Raft algorithm,as the number of nodes in the distributed storage system increases,the leader node will incur significant log distribution overhead and require more time to distribute logs to other nodes in the cluster for log item submission,which greatly affects system performance and limits the horizontal scalability of the distributed storage system.To solve the limitation of Raft algorithm in horizontal scalability in distributed storage systems,this thesis proposes a Double-Raft distributed storage solution mainly composed of a consistency hash cluster based on the Raft algorithm and a storage cluster based on the Raft algorithm,supporting high concurrency,massive storage,and high reliability.The main work of this thesis is as follows.Firstly,in order to overcome the limitations of horizontal scalability of the Raft algorithm in the context described above,this thesis proposes a Double-Raft distributed storage scheme.In this scheme,a consistency hash cluster divides the files managed by each storage cluster,and each storage cluster independently maintains its own file information.The storage clusters operate independently of each other,and the system can achieve horizontal scalability by adding a new storage cluster in the consistency hash cluster and completing the migration of relevant data.Secondly,during the log distribution process in Raft algorithm,a mechanism based on dynamic priority allocation was proposed by this thesis to allow the leader to determine the order in which logs are distributed to follower nodes based on the synchronization status between the leader’s logs and the logs in the follower nodes.This enables faster replication of log entries to more than half of the nodes in the cluster,accelerating the submission speed of log entries,and improving the system’s write request throughput.Finally,during the log distribution process led by the leader of the Raft algorithm,a delegation mechanism based on windowed pipeline was proposed by this thesis.This mechanism allows the leader to assign followers with higher synchronization levels to distribute logs to followers with lower synchronization levels,thereby shifting the burden of distributing logs from the leader to the followers.As a result,the mechanism reduces the time for node data to converge.