Research On The Optimization Of Distributed Key-Value System Based On Redis

As an important tool to improve the data access speed of data centers or applications,distributed Key-Value systems,such as Redis and Memcached,etc,have become an important part of the critical infrastructure in many data centers.Their performance will directly affect the QoS of many large-scale services.Therefore,it is of great significance and value to improve the performance of the distributed Key-Value system.In recent years,it has also received extensive attention from the industry and academia.Distributed Key-Value systems typically use a relatively simple data model and provide a quick lookup capability similar to hash tables,but these systems often have performance bottlenecks in network processing,parallel control,or Key-Value data processing,etc.In recent years,both industry and academia have spent a great deal of effort to improve their performance.Redis,one of the most popular open source Key-Value systems,is currently the most used system in all Key-Value systems.However,Redis still has shortcomings in distributed cluster solutions and network processing,etc,leading to the loss of Redis performance.In view of this,the main innovative work of this paper is as follows:(1)Based on Redis,this paper studies and analyzes the deficiencies of the commonly used data sharding scheme for Redis clusters,and proposes an optimized data sharding solution based on Zoo Keeper that supports cluster dynamic scaling and data automatic migration.The solution uses ZooKeeper to provide a reliable consistency services to uniformly maintain the cluster's data sharding information,removes the proxy layer used by existing mainstream solutions,and optimizes the client to obtain a Redis cluster solution with higher throughput and lower latency.(2)Based on the optimized data sharding solution based on ZooKeeper,this paper further studies the high-availability scheme of distributed Redis clusters,and proposes a distributed Redis cluster high availability solution based on Zoo Keeper and Redis-Sentinel.The solution uses Redis' s replication capability to start multiple secondary nodes to replicate the primary node and uses Redis-Sentinel clusters to monitor the status of the primary and secondary Redis nodes.When a primary node fails and goes offline,Redis-Sentinel will select one of its secondary nodes as the new primary node,and notifies the client of the switch information through Zoo Keeper,thus realizes the automatic failover of the cluster and ensures the availability of the cluster.(3)For the characteristics of the distributed Key-Value system that needs to deal with a large number of small requests,this paper also studies and analyzes the deficiencies of the Redis network I/O model when dealing with a large number of small requests,a nd proposes and implements the pipelining and other batch operations to further optimize the query performance of the Redis cluster.The traditional network I/O model adopted by Redis has a large number of overheads,such as system calls,context switching,and interrupts,etc,when processing a large number of small data packets.Through the batch operation such as pipeline,the Redis client can send multiple commands to the Redis cluster at one time,thereby reducing the network overhead of the Redis cluster and further improving its performance.Finally,this paper tests and compares the performance with the current mainstream proxy-based Redis cluster solutions,and verifies that the proposed optimization scheme not only guarantees the cluster's high availability,but also supports the cluster's dynamic scaling and data automatic migration.At the same time,the performance of the cluster has also been greatly improved.The throughput of the cluster is up to 200% more than the current mainstream Redis cluster solution,and the latency is reduced by more than half compared to the mainstream proxy-based solution.