Research On Disk Fault Prediction And Early Warning

With the rapid expansion of data center storage system and the vigorous development of cloud computing,blockchain,and other technologies,the amount of data generated has increased explosively.In this era of information and digitalization,the disk is the main carrier of data storage.Therefore,the safety and reliability of its operation have become the focus of many enterprises.Due to the particularity of its structure,the disk may be irreversibly damaged in case of failure,resulting in the loss of all the information stored in it,which will bring great economic losses to the enterprise.Although the quality of the disk can be evaluated by manual detection or threshold definition method,its high cost and low classification accuracy do not allow these methods to be applied to realworld problems.Thus,disk failure prediction has become the main way to solve such problems.Although the disk has a SMART defense mechanism,it only relies on the original monitoring data to predict the disk failure,which leads to low accuracy.In the actual disk data set,the research on disk failure prediction needs to be carried out after the statistical analysis of SMART attribute data.The existing disk failure prediction based on SMART attribute data still has some problems,such as small data scale,relatively balanced positive and negative samples,low data dimension,unpublished experimental data set,etc.According to the characteristics of the disk data set including large samples,high latitude,and extreme imbalance of positive and negative samples,a disk failure prediction model based on features with high correlation coefficient is proposed in this paper.Many comparative experiments are conducted in the actual open data set to verify the model’s validity.The main work of this paper is as follows:(1)Analyze the disk model and data distribution in the disk public data set,screen the target types of disks according to the monitoring report and data analysis,and complete the data cleaning.Select the features with a higher correlation coefficient with disk failure based on the correlation coefficient matrix in order to reduce the dimension of the data and make the model easier to fit.Regarding the imbalance between positive and negative samples,the under-sampling strategy is realized by mini batch Kmeans clustering.Three data sets with different positive and negative sample ratios are obtained to support the prediction of the machine learning model.(2)This thesis proposes a disk fault prediction method based on improved LightGBM model.Aiming at the characteristics of massive,high-dimensional,and unbalanced positive and negative samples of disk data sets,the LightGBM model with high prediction accuracy and fast training speed for large-scale data sets is selected,and the model is improved for the imbalance of positive and negative samples.Focal loss function is introduced to reduce the impact of positive and negative sample proportion imbalance on prediction.To verify the advantages of the algorithm in solving such problems,the effectiveness and efficiency of the improved LightGBM model in disk fault prediction are highlighted through the comparison of multiple indicators and training time with the training results of four traditional integrated learning models in data sets with different positive and negative sample ratios.(3)This thesis proposes a disk fault prediction method based on abnormal fraction time series characteristics.The whole original data set is processed by anomaly detection method,and the disk fault prediction is realized by machine learning method on the new data set composed of anomaly score sequence characteristics.According to the characteristics of the disk data set,the isolated forest algorithm is selected to calculate the anomaly score of the full data efficiently,sort the obtained anomaly score,and formulate an appropriate anomaly threshold to distinguish between abnormal data and normal data in combination with the number and proportion of fault samples in the data set.Retain the abnormal scores of the abnormal data obtained through screening,replace the abnormal scores of normal samples,and finally get three groups of data sets with new features composed of abnormal scores of different numbers of samples.Through the previous research,the improved LightGBM model with the highest prediction accuracy is selected to predict the disk fault of three groups of data sets,compare the prediction results of the model in the abnormal score sequence feature data set with unbalanced proportion of positive and negative samples,compare multiple prediction result indicators,and screen the optimal number of abnormal score sequence features in the disk fault prediction problem,which verifies the effectiveness of this method.Finally,the paper compares the fault prediction results of the two methods under the same data environment,and makes a comparative analysis combined with the experimental results and the real environment.