Design And Implementation Of Multi-Replica Replication Protocol Based On RDMA And Persistent Memory

In recent years,with the rapid developing of remote direct memory access(RDMA)and persistent memory(PM),because of the low persistence overhead of PM and the high performance of RDMA,they are widely used to accelerate the data replication in distributed replication protocols.However,in order to ensure strong consistency of local reads,existing replication protocols often require all replicas to obtain a complete data copy before the update is committed,which increases the replication overhead on the critical path and limits scalability.In addition,with the maturity of persistent hardware technology,a small number of critical data copies can provide sufficient failover redundancy.How to rebalance the relationship between performance,failure reliability and strong consistency guarantee of replication protocols with new hardware becomes a new challenge.In order to solve the shortcomings of existing replication protocols,an asynchronous replication protocol based on RDMA and PM with decoupling of metadata and data is proposed,which includes two key contents.Firstly,the protocol divides replicas into a leader node pair and a replica chain.The leader node pair uses PM’s low persistence overhead feature to perform fast synchronous replication and provide fault recovery guarantees.The replica chain is organized in a chain topology,received metadata broadcasts from the leader node pair,and equipped one-sided RDMA communication for asynchronous replication with high performance and low CPU consumption through a memory-aware data placement strategy.The replica chain provides additional data redundancy as well as loadbalanced local reads.Secondly,in order to meet the design requirements of the replication protocol with high throughput and low latency,a high-performance multi-threaded data replication framework based on RDMA and PM is proposed.The replication framework implements an efficient huge page memory management mechanism,a high-performance multi-threaded RDMA communication framework and a concurrency control mechanism for shared data access and partitioned data access.In order to reduce the CPU consumption of replica nodes,bulk RDMA message confirmation optimization is implemented.Moreover,a cache-friendly multi-threaded message queue mechanism is designed to ensure the efficiency of multi-threaded replication.Experimental results show that this decoupled asynchronous data replication protocol,implemented in a high-performance multi-threaded replication framework,can reduce the write latency under a specific workload by 83% and the average write latency is reduced by71%,compared to the state-of-the-art replication protocols.Under the 16 KB largegranularity workload,the throughput is increased by up to 3.5 times,and the average throughput is increased by 89%.At the same time,the protocol has good replica horizontal and multi-thread scalability.Analysis and discussion derived from experimental results can also provide replication applications with guidance about how to choose appropriate replication protocols.