Design And Implementation Of Distributed Graph Database Storage Engine Acceleration And Transaction Management

In recent years,distributed graph databases have been growing in popularity,especially in areas such as recommendation systems,financial risk control,social networks and knowledge graphs,where the value of relationships in graph data is particularly valued.However,distributed graph databases for online analytics scenarios still face performance problems such as large network communication overheads in the computation and storage layers and poor system parallelism.Therefore,how to improve the performance of online query and online analysis poses a new challenge to the design and implementation of distributed graph databases.Based on the mainstream compute-storage separation architecture,this thesis focuses on the acceleration strategy and transaction management for graph storage engines,proposes an asynchronous distributed graph traversal algorithm based on computational pushdown,various acceleration mechanisms for graph storage engines,and designs an asynchronized distributed transaction to meet the graph query workload under large-scale graph data.The main work of this thesis and the innovation points are as follows.1.In terms of algorithm design,this thesis proposes an asynchronous distributed graph traversal algorithm based on computational push-down.The algorithm takes full advantage of the continuous local execution of the circle extension at the storage nodes and reduces the network communication overhead between the computation and storage layers.Moreover,the algorithm adopts an asynchronous design at both levels of single-computer execution and multi-computer collaboration,and optimizes the efficiency of multi-threaded parallelism within a single machine and task parallelism among multiple machines through a two-level scheduling mechanism to improve the performance of online analysis of distributed graph database systems.2.In terms of acceleration strategies,this thesis proposes a variety of acceleration mechanisms for graph storage engines.The logical isolation of computation and I/O divides threads as resource groups and the logical partitioning of the execution of graph traversal tasks improves the utilization of single CPU resources and takes advantage of the asynchronous execution of computation and I/O.The multithreaded task-stealing scheduling avoids task load imbalance among threads and reduces the long-tail latency of the system.And the computational bloat within the operator is controlled by vectorized execution,which slows down the I/O pressure of short-time large data batches.3.In terms of transaction processing,this thesis designs an asynchronized distributed transaction.The distributed transaction optimizes the classical two-stage commit,and optimizes the mark of successful transaction commit from requiring all participants to successfully apply transaction modifications in the classical scheme to requiring only one participant to successfully write a transaction status record,and the transaction modifications are applied actively by asynchronous threads or piggybacked by other transactions.By applying transaction modifications to the original data asynchronously,the processing efficiency of distributed transactions is effectively improved.In the testing phase,this thesis provides a complete test of the functionality and performance of the graph storage engine,and a detailed analysis of the optimization strategies based on latency and throughput.It is proved through experiments that algorithm optimization,system optimization and transaction optimization can effectively improve the performance of the graph storage engine and the system as a whole for online query and online analysis.