An OLAP-Oriented Distributed Key-Value Storage Engine

Nowadays, the amount of data produced by enterprise information systems increases rapidly. The traditional On-Line Transaction Processings systems for enterprise information management are unsatisfactory. In this context, Data Warehouse has achieved great development. On-Line Analytical Processing which is refered as OLAP is one of most important application technology of Data warehouse. With the advent of the age of big data, OLAP systems have been widely used. OLAP systems focuses on providing decision support to decision-makers and making complex querying processing according to their requirement. Meanwhile, OLAP provides the querying results to decision-makers in an intuitive way in order to assist them with understanding the needs and developing the proposal. As the basis of an OLAP system, the storage subsystem plays a key role. Its throughput, access delay, availability and scalability will directly affect the scale of data and performance for an OLAP system as well as the timeliness and efficiency for results obtained by querying processing further.This article describes the design, technology and implementation of a distributed key-value storage engine which is named T-IMDB. T-IMDB has several innovation First, T-IMDB keeps all data in DRAM while disk is only used for backup. It employs log-structured memory and two-stage cleaning which reduce fragmentation effectively and help T-IMDB maintain high throughput and low latency under high memory utilization. Second, a scalable distributed index is introduced to support range querying on the key-value data model which is consistent with data within a limited time window. Third, T-IMDB provides strong consistency for their replicas and this is convenient for developers to implement business logic. Meanwhile, external system can use T-IMDB smoothly with the help of strong consistency.T-IMDB is implemented based on the asynchronous event-driven networking programming model. At the same time, it applies the SOA thought for its design which decouples the network traffic and the business logic. The experiments in this article also shows that T-IMDB can perform the desired function correctly and has excellent performance.