Research And Implementation Of A Disk Failure Prediction Tool Based On Multi-source Data

With the rapid development of cloud services and the increase of data scale,the demand for data storage infrastructure has increased.Today’s cloud server providers usually deploy data centers,which are equipped with millions of disks all over the world,and disk failures are unavoidable.There are two measures to deal with disk failure:redundant backup and advance prediction.The latter can better avoid the loss of disk failure and save costs.At present,most of the research on disk failure prediction is at the academic level.The failure prediction model is single,and the accuracy rate is low for heterogeneous disks.The model training and application are based on machine learning platform,and there is a lack of visual interactive tools for directly predicting disks using the failure prediction model.In view of the above problems,this paper designs and implements a disk failure prediction tools based on the actual requirements and the multisource disk data of real data centers.(1)In view of the problems of large scale,different structure and redundant storage of multi-source disk data in the data center,in order to better monitor the disk status and use the disk data for failure prediction model training,this paper designs a multi-level storage architecture of collection-process-storage to realize the collection and storage of disk SMART data,performance data and server data.(2)In view of the problems of insufficient adaptation of single failure prediction model in heterogeneous disks,model performance declining over time,and model being limited by platform,this paper conducts feature engineering on disk SMART data and builds disk failure prediction model,encapsulates the model and provides prediction interface,supports model uploading and loading,provides multiple prediction models for users to choose flexibly,and realizes model management and application.(3)In view of the lack of practical application of disk failure prediction model and the lack of intelligent tools for disk management in data centers,this paper designs a prediction tool with simple operation,batch processing and data visualization,which integrates disk failure prediction and status monitoring.The monitoring panel for real-time monitoring of disk performance data is implemented,providing one-key failure prediction for flexible selection of disk and model,and early warning and prompt for risk conditions to help operation and maintenance personnel manage and make decisions.