Design And Implementation Of Message-oriented Middleware Storage Engine Based On Non-volatile Memory

Message-oriented middleware is an essential component in the back-end system.With the increase of data volume and the retardation of memory capacity in the current era,in the extreme scenario of continuous data mixed read and write,the cache pollution caused by the lack of cache will seriously drag down the performance of the middleware.Rather than volatile memory(persistent memory),its excellent performance,relatively low price,byte-addressable characteristics and these advantages of persistence,to the message middleware storage has brought a new choice.To give full play to the advantages of non-volatile memory,it is necessary to redesign the storage system of middleware.Different from other systems,message middleware has its own unique data characteristics: message is a large number of small capacity data;The messages in the same queue are read in sequence.Messages cannot be consumed repeatedly,and all messages are read only once.These characteristics bring challenges to the introduction of non-volatile memory middleware storage systems.Based on non-volatile memory,this paper designs a middleware storage engine based on non-volatile memory according to the data characteristics of message-oriented middleware.Specific work and innovation points are as follows:1.This paper designs a sequential write and random read message storage scheme,which can effectively play the non-volatile memory random read speed,and adopts multi-thread aggregation synchronous disk flushing strategy,which can ensure data power safety under the premise of full SSD bandwidth.2.By maintaining an index in the memory,this paper realizes the two-layer cache architecture design of the memory and non-volatile memory and unified management of different storage media,reducing the dependence on page cache.3.Through the design of non-fixed-length cache block,hierarchical cache pool and cache threshold,this paper effectively improves the reuse efficiency of cache and enables the cache with limited capacity to provide more read services.Optimizations such as 4K alignment,message preloading,file prereading,and partial cache of large message bodies can effectively improve the performance of middleware and reduce IO amplification.Finally,experiments were carried out in this paper,and the test data showed that the write speed could reach the upper limit of SSD bandwidth,and the read performance and throughput in mixed read and write scenarios were improved by 2-15 times compared with the storage system using only memory,and also improved by 20%-60% compared with the scheme using non-volatile memory as high-speed hard disk.