Research And Implementation Of Distributed Consensus Algorithm For Erasure-coded Storage System

In recent years,with the rapid development of the Internet era,the surge of user data has brought great challenges to storage technology.Most of these data are infrequently accessed,which are called cold data.The storage of cold data does not require high read performance of the system,so special low-cost equipment and erasure coding fault tolerance technology are usually used for storage.However,some of the current methods for storing cold data are cumbersome and inefficient,and some have consistency problems.Therefore,this thesis designs a cold data storage algorithm,which can provide higher storage efficiency and consistency guarantee for cold data storage.Cold data storage often uses erasure-coded redundancy mechanism because it is a fault-tolerant mechanism with low storage overhead.Therefore,the algorithm studied in this thesis is also based on erasurecoded data storage.There are two traditional erasure-coded data storage methods:asynchronous or synchronous.For the synchronous storage method,data is encoded as it is written to the server,and the writing is successful only after all these coded-fragments are stored.This method underperforms in availability because the storage failure of any codedfragment will disturb the writing process.For the asynchronous storage method,the system stores the data first in the form of copies,and then asynchronously completes the encoding process and deletes the copies.This storage process will bring unnecessary disk IO and network traffic overhead.Some consensus protocols such as RS-Paxos and CRaft,which are based on Paxos and Raft,can solve the above problems.However,RSPaxos cannot achieve the same liveness as Paxos.CRaft cannot achieve the lowest storage and network overhead.Therefore,to better solve the problems of synchronous and asynchronous methods,this thesis studies and implements a consistency algorithm ECRaft that can be used for cold data storage.This algorithm is based on Raft,and it can also achieve the same fault tolerance as Raft.Storage space utilization can reach the best level that erasure-coded storage systems can achieve.We build a key-value store based on ECRaft to evaluate it.In our experiments,compared with CRaft using complete-entry replication,ECRaft can save 63%of the storage,increase write throughput by 28.2%,and reduce write latency by 19%.In order to verify the performance of ECRaft in actual systems,this thesis also implements a distributed storage system based on ECRaft and proposes the architecture of Multi-ECRaft.The system is divided into three modules:client,metadata management unit,and storage unit.Experiments show that the system using ECRaft is better than the system without it in terms of write throughput,write latency and availability.