Energy Efficiency Optimization For Hadoop Clusters

Widespread use of computer clusters has brought people convenience, and it alsobrings energy consumption problems. How to use energy resources efficiently has becomea key problem to be solved. As the most popular software framework for processing oflarge data sets, Hadoop is deployed on more and more clusters. But Hadoop is developedwithout energy efficient features. So it is necessary to do some research on energyefficiency optimization for Hadoop clusters.Based on existing research achievement, combined with the rack-awareness featureand replica placement strategy of Hadoop, the solution that reconfigures the clusteraccording to the workload is proposed. Two modules are added to the Hadoop cluster: thecluster workload monitor module and the cluster reconfiguration module. The workloadmonitor module is responsible for monitoring the average cluster utilization, and if it findsthe utilization is too high or too low, it will inform the reconfiguration module toreconfigure the cluster. The reconfiguration module contains two algorithms: the retiringnode algorithm and the resuming node algorithm. When the average cluster utilization istoo low, a number of active nodes will be retired by the retiring node algorithm to improveenergy efficiency of the cluster. Also when the utilization is too high, a number of inactivenodes will be resumed to improve performance of the cluster.The proposed algorithms are evaluated by simulation experiment using GridSimtoolkit. The state of the cluster with energy efficient strategy is compared with that of thecluster without energy efficient strategy. As the results show, the proposed algorithms canactually reconfigure the cluster according to the workload. The energy savings denoted bythe experiments is more than30%.