Design And Implementation Of Key Value Storage System Based On Multi-Node Dynamic Partition Mode

With the rapid development of mobile internet and the Internet of Things around the world,the huge scale of internet users has generated massive amounts of data.In order to store and process this data,people are focusing on developing more efficient storage systems.The key value storage system stores data in the form of key value pairs and has strong scalability.On this basis,a multi machine key value storage system stores key value pair data on multiple computer nodes,providing greater throughput and storage space.Therefore,multi machine key value storage systems are widely used in large-scale applications as storage layer subsystems.However,with the increasing amount of data storage and the faster growth of data volume,the problems of complex expansion operations and difficult maintenance when the storage volume is too high in traditional multi machine key value storage systems are gradually exposed and urgently need to be solved.Therefore,the topic of the paper has practical application significance.This article analyzes the shortcomings of traditional key value storage schemes and applies the LSM tree storage structure to design and implement a key value storage system based on multi machine dynamic sharding mode.It supports accessing databases in the form of APIs and reduces usage costs.Firstly,this article designs and implements a data dynamic routing scheme for the entire reading and writing process,ensuring that data reading and writing operations can be carried out correctly in a distributed deployment environment;The application of message queue has designed and implemented a dynamic sharding expansion scheme that does not require business interruption,achieving transparency and no perception of users throughout the expansion process.Then,this article proposes and implements a snapshot generation algorithm based on priority queues and fine-grained locks to address the problem of spatial amplification in LSM trees.Finally,based on the above solutions and algorithms,this article designs and implements a key value storage system based on multi machine dynamic sharding mode.And conducted requirement analysis,preliminary design,detailed design implementation,and testing for the system.This article conducted functional and performance tests on the implemented system,and the test results showed that the system can adapt to distributed scenarios,expand in a sharded manner,and operate conveniently.The expansion process is transparent and insensitive to users,and effectively reduces the impact of spatial amplification.The test results indicate that the technical route of this article is feasible and has practical application value.