Design And Implementation Of A Distributed Storage System Based On RocksDB Engine

With the development of the information society,the Internet has entered a new era with rapid development,and the amount of data on the Internet has exploded.With the continuous increase of data volume,the traditional relational database has been unable to cope with the application of the current massive data scale,and the NoSQL database has emerged.In recent years,the development of NoSQL technology has come a long way.Various NoSQL databases have emerged one after another,including column databases,graph databases,document databases,and key-value databases.The key-value database is the most important branch of the NoSQL database.The storage engine is an important part of the database,and its performance directly determines the speed of reading and writing for database storage systems.The current LSM-Tree based storage engine,leveldb,with better write performance,and the LSM’s design avoids a large amount of random writes;it also achieves good performance for specific reads,that is hot data in memory,and Facebook maintains an active LevelDB branch called RocksDB.RocksDB has made a lot of improvements in LevelDB,such as multi-threaded Compactor,hierarchical custom compression,multiple MemTable,and so on,which performs better than leveldb.This thesis mainly has three aspects of work:(1)First of all,it summarizes the emergence and development of NoSQL technology,and through the understanding of the research status of storage engines and distributed NoSQL storage at home and abroad,analyzes and compares the existing storage engines,summarizes their respective advantages and disadvantages and uses scenarios;analysize the design principles and features of the RocksDB storage engine,which has extremely high write performance.(2)Deeply study the theory and related technologies needed to implement distributed storage systems,and then design a distributed storage system node layout architecture,which can flexibly add and delete data nodes and quickly locate the data location.Analyze the applications of rocksdb,apply the rocksdb storage engine to distributed storage systems,improve the distributed storage system architecture,then design and implement a distributed key-value storage system to ensure high read and write performance and high availability.(3)Perform in-depth functional and performance test of the distributed distributed key-value storage system,then analyze the test results in detail,and compare it with the performance of Redis,a high-efficiency key-value storage system widely used today.The test results show that this system has little difference with Redis in terms of write performance and has reached a good level,which proves that the system achieves the expected goal.Finally,it summarizes the work of this thesis and looks forward to the future work..This thesis designs and implements a distributed key-value storage system based on RocksDB storage engine.It has good exploration and practical significance for the application of storage engine in the field of distributed storage system.