Research On The Cloud Storage Technology Based On Heterogeneous Hadoop Under Vehicular Environment

Internet of Vehicles is increasingly affecting our lives and it contains massive valuable data, hence the cloud computing in Internet of Vehicles is becoming more and more significant. As a leader in cloud computing, Hadoop is increasingly popular in big data. The hardware environment of the Hadoop cluster is generally homogeneous under the initial conditions, however, with the passage of time, the homogeneous environment will gradually evolve into a heterogeneous environment and the performance of storage policy will be reduced. Therefore, a storage policy based on data temprature and node performance is propsed in this thesis to slove this problem.Firstly, in order to obtain the node performance in specific Hadoop application scenarios more accurately, a node performance calculation method based on multivariable linear regression model is studied. This method uses benchmark performance measurement tools to obtain the read and write speed of disk, random access rate of memory and process power of CPU, then establish the linear regression model between node performance and three parameters to calculate estimated performance of Data Nodes. The experimental results show that the node performance in the experiment is consistent with the calculated value which verifies the feasibility of the model.Secondly, in order to improve the storage performance of hetereogenous Hadoop clusters, a storage strategy based on node performance and data temperature is studied. On the basis of hetereogenous storage scheme of Hadoop distributed file system, the difference in performance between nodes of the same type storage media are considered. By establishing the mapping relationship between data temperature and storage policy, files which are frequently accessed and recently visited are moved to nodes whose performance is better, thereby achieve "An able man is always busy" in the cluster. Experiments show that this method has a considerable improvement in the read operation performance of the vehicle networking cloud platform which is deployed in Hadoop cluster.