The Implementation Of Data Migration Mechanism For Hierarchical Storage

Hierarchical storage is designed to realize that, the data with different heat valueshould be on the storage devices with different performance, maximize storage utilization.When the heat value of the data is changed, it will trigger data migrate to the storagedevices which are better meet their current heat value. Therefor, for hierarchical storagesystem, design a data migration mechanism which is transparent and efficient toforeground application has important research value.It divided into three modules to design data migration process: heat statistics module,strategy matching module and data migration module. The heat statistic module isdesigned in the generic block layer to count access frequency, and design a heat evaluationmethod base on access frequency and time weighting value. The strategy matchingmodule build client-specific policy rules and common to all clients policy rules, anddesign a threshold dynamic adjustment algorithm based on Physical Volume (PV)occupancy and an algorithm based on the load balancing of storage nodes to apply PV.The data migration module design three kinds of data migration algorithms, which canachieve re-migrate at breakpoint:(1) the simple data migration algorithm (SCP), just readthe data out from source volume then write to destination volume;(2) the algorithm baseon the kernel copy thread (kcopy) in Device-mapper, which implement data consistency;(3) the algorithm base on SCP, in which read process and write process can executeparallel. In order to protect the data consistency in data migration, we design and analysistwo kind of consistency methods: a method based on record-write strategy, an othermethod based on piecewise processing.Finally, concrete realization of a prototype of the data migration mechanisms, and dosome test to data migration, focus on the speed of data migration, and the impact to usersin data migration progress. Experiment indicate that, in the case of high load, the diskthroughput decreased by10%to25%in data migration; Under normal circumstances,data migration has little affect to users when they access the system.