Design And Implementation Of High Availability Solution For Hadoop

As rapid development of Internet brings the data exlosion, plenty of enterprises use distributed file system to build their high performance and scalable applications to support huge volume data storage and analysis. Hadoop is one of the most popular open source distributed file system, which provides high scalability and strong fault tolerance, can run on commerical computers. However, Hadoop does not support high availability yet. Because Master/Slave architecture is adopted in Hadoop and NameNode is the Master which turns to be a SPOF (Single Point of Failure). This paper proposes a hot-standby and automatic-failover solution HANC (High Availability Node Cluster) to remove SPOF. Implementation and experiment results are also presented. The crucial contributions are as follows:1. We introduce the architecture of HDFS (Hadoop Distributed File System), analyze main feature of NameNode and DataNode including metadata, startup process and SafeMode state. Besides, we give the concept of high availability technology. Based on analysis of high availability technology for software system and hardware system, we introduce the high avalibility technology for distributed file system including state monitoring, data replication and continuous service.2. We consider NameNode is the SPOF of Hadoop and study current solutions for Hadoop high availability. According to backup mechanism and switch type, we classified those solutions into five categories and each one’s implementation details are presented in this paper.3. We present the architecture of HANC which is composed of three components:data synchronization component, cluster management component and client access component. Specifically, data synchronization component includes metadata synchronization, block location synchronization and lease management. Cluster management component takes responsibility of multiple NameNodes and failover process. Client access component redirect client to a new NameNode in case of primary NameNode failure. Besides, a mechanism of improving HDFS read throughput is proposed.4. The obtained results by our experiment show failover time is in a statble value about 6 seconds. In contrast with Hadoop0.20.2, the delay time in HANC has been reduced and throughput is increased, which demonstrate feasibility and efficiency of our solution.Conclusions and future work are discussed in the end of this thesis.