Research And Application Of Adaptive Learned Index Technology For Distributed Storage System

In large-scale distributed storage systems,index has always been one of the core structures of distributed storage systems,playing a crucial role in accelerating data access.In the era of rapidly growing data volume,traditional index have a high demand for memory space and have reached the limit of optimization in terms of performance.How to solve the problem of large space occupation and index performance degradation caused by massive data on index structures is a very important research direction in the field of distributed storage.The storage engine,as the most core component in distributed storage systems,is the best entry point to solve indexing problems.This article starts with storage engine LSM-Tree,a commonly used storage engine in distributed storage systems,as an example to study and apply adaptive learned index.The following results have been achieved:1.Study the feasibility of applying learned index in the distributed storage engine LSM-Tree,and prove that learned index can be applied in the SSTable metadata indexing process and file indexing process in LSM-Tree.2.On the basis of the above feasibility study,this paper studies and proposes a learned index model in the SSTable file,and supports persistence to the disk.We have studied and proposed an adaptive learned index construction method,which enables the learned index to adaptively guide the construction of SSTable learned indexe based on historical information.In performance comparison experiments on real datasets,compared to traditional index,SSTable learned index occupy less disk space and significantly improve query performance.3.A Merge PLR recursive level learned index model for indexing SSTable metadata was proposed,which supports dynamic key insertion through a buffer.When the buffer is full,it is merged with the constructed learned index model.In performance comparison experiments on real datasets,Level learned index occupy less disk space and improve query performance compared to similar learned index.4.Developed an adaptive learned index based log structure tree(LIA-LSM)storage system.This storage system can serve as the underlying key value storage engine,providing interfaces such as Put,Get,and Delete for upper level applications.It can support multiple application channels such as databases as applications and standalone storage engines for distributed storage systems.