Research And Implementation Of Data Distribution Method Based On Logical Data Entity In HCOSS Object-based Storage System

Object-based storage is an emerging storage method,which has been widely used in large-scale data storage and cloud storage fields.In object-based storage,the quality of data distribution directly affects system performance,so how to reasonably distribute data objects to storage nodes is an important research direction.Based on the analysis and research of the current data distribution methods,distribution strategies and algorithms in object-based storage,and based on the object-based storage system HCOSS(Hyper-Converged Object-based Storage System)independently developed by the laboratory team,this paper designs a data distribution method with a triple mapping relationship is implemented and applied in HCOSS.First of all,in order to optimize the HCOSS system architecture,the File Service Gateway is proposed.On the file service gateway,the unified entry of user files,separate storage of file metadata and data content,file pre-processing and post-processing,and data object pre-distribution are studied and realized.Through the file service gateway,the direct interaction between the user and the storage system is reduced,thereby reducing the system burden and improving the read and write performance of the system.Secondly,in order to establish the intermediate mapping unit of HCOSS data distribution,the Logical Data Entity is proposed.Through the logical data entity,the data objects corresponding to the associated small files are reasonably organized and stored,which optimizes the local storage of small files;The entities that are organized together by programs associated with data,this design greatly reduces the frequency of access to the metadata server,enabling HCOSS to achieve decentralization to a certain extent.Thirdly,this paper takes the logical data entity as the intermediate mapping unit of data distribution,and designs a data distribution method with triple mapping relationship.In the data distribution method,the object uniform distribution strategy,the object load balancing strategy and the multi-copy strategy are studied and realized.By improving the existing jumping Hash algorithm,a LDE＿Hash algorithm deeply integrated with HCOSS is proposed,and based on this algorithm,the dynamic uniform distribution strategy of logical data entities is realized.The flexibility and adaptability of the HCOSS system are obviously improved by triple-mapping.Finally,this paper builds an experimental environment by deploying virtual machine clusters on physical machines,and conducts detailed tests on various data distribution strategies,file service gateway functions,and node scaling and self-adaptive aspects of the HCOSS system.The results show that the data distribution method implemented in this paper meets the actual data distribution requirements of HCOSS,and has a certain value of promotion,application and further research.