Collaborative Optimization Of Transaction Protocol And Consistency Protocol In Distributed Databases

The advent of mobile computing,cloud computing,and artificial intelligence has resulted in the expansion of data and user scale,coupled with variable business scenarios such as hotspots and burst traffic.In response to this,traditional commercial databases are no longer sufficient to meet the rapidly growing business demands,prompting a shift towards distributed architecture.In distributed databases,transaction processing,which serves as a critical data management function,is presented with a multitude of challenges,including strong consistency,high throughput,high availability,and scalability.Typically,distributed databases employ data sharding to achieve horizontal scalability,while simultaneously maintaining multiple replicas for each shard to ensure high availability.Consequently,how to achieve high-throughput and scalable transaction processing while ensuring strong consistency of transactions through efficient inter-node coordination has become a valuable research topic.Transaction processing in distributed databases necessitates the cooperation of two protocols: the transaction protocol,which ensures serial scheduling and atomic commit of transactions,and the consistency protocol,which maintains the consistency of state among replicas.The two protocols will result in multiple rounds of network communication,significantly increasing the latency of transactions and affecting system concurrency.In this paper,we investigated the interactions between transaction and consistency protocols and identified issues and optimization opportunities in their collaboration.Furthermore,we explored more efficient collaboration models by working in three directions,i.e.,integrating the transaction and consistency protocols,optimizing consistency protocol via transaction characteristics,and optimizing transaction commit by using leaderless consistency protocol.Our goal is to optimize the collaboration model of both protocols and address the challenges inherent in distributed databases.The primary contributions of this paper are as follows:(1)We combined transaction protocols with consistency protocols to design an efficient multi-master transaction processing mechanism for cloud database with disaggregated-storage architecture: In cloud databases,the separation of storage and computing layers enables the scalability of computational resources through adding nodes to the computing layer.To fully leverage the elastic scaling capabilities of the computing layer,it is crucial to support the concurrent execution of transactions across multiple computing nodes.To this end,the design of an efficient transaction commit protocol is essential to synchronize transaction results between the computing and storage layers.In this paper,we proposed a multi-master transaction processing mechanism that combines transaction protocols and consistency protocols to enable fast decentralized committing of non-conflict transactions within one round of communication.Besides,the proposed transaction commit protocol minimizes the performance degradation caused by conflicting transactions among multiple computing nodes by centralized reordering conflicts,thereby ensuring precise conflict resolution.(2)We optimized the consistency protocol by considering the characteristics of transactions,and proposed a parallel replication Raft protocol for transactional systems: Although Raft has been widely used in the construction of highly available database systems,Raft does not consider the specific requirements of applications,such as highly concurrent transaction processing in databases.As a result,the serialized log replication of Raft can limit the performance of transactional databases.In this paper,we designed the DP-Raft(Dependency Preserved Raft)protocol dedicated to transactional systems.The DP-Raft protocol addresses the different requirements of different transaction dependencies on the order of log replication.By relaxing the serialization restriction of Raft and supporting parallel log replication,the DP-Raft protocol compensates for the performance gap between transaction processing and log replication,thereby improving the overall performance.Moreover,the protocol is aware of the dependencies of transactions and maintains the replicate order among dependent transactions,thus satisfying the requirements of transaction semantics.(3)We optimized distributed transaction commits by adopting the leaderless consensus protocol,and proposed the decentralized commit coordination mechanism for geo-distributed databases: Geo-distributed databases achieve regionallevel disaster recovery and are able to serve users in close by deploying replicas in multiple regions around the world.However,distributed transaction commits encounter significant challenges due to the inherent communication overhead involved in cross-region network communications that are caused by two-phase commits and consistency protocol-based log replication,both of which lead to increased transaction commit latency and longer concurrency control period,and affect system concurrency.To address these issues,we presented an in-depth analysis of the causes of high transaction latency and long concurrency control period in existing geo-distributed databases.Then we used the leaderless consensus protocol to maintain a set of coordinators in multiple regions across the world,where each coordinator makes transaction commit decisions independently in a decentralized manner and notifies clients and participant leaders nearby,thus avoiding the communication overhead with remote coordinators in the traditional single coordinator model.At the same time,the participant leaders can end the concurrency control period as early as possible once they are informed of the commit decision from the nearby coordinator,thus achieving higher concurrency of transaction processing.In summary,this paper aimed at improving the performance of transaction processing by optimizing the collaboration mode between transaction protocols and consistency protocols in multiple scenarios from three aspects,thus achieving the effect of strong consistency,high throughput,high scalability,and low latency of transaction processing.By conducting extensive experiments,the effectiveness of the proposed mechanism and protocols are verified.