Research On Heterogeneous Virtual Machine Platform Of Supporting Hadoop Configuration

With the development of cloud computing technology, data centers with various sizes have been appearing, but these data centers often with various virtual machine management platforms (e.g. Eucalyptus, OpenNebula, OpenStack), completely different demand scenarios, different management platform requires different maintenance, development skills and experiences, while server resources cannot be shared dynamically between different management platforms, affecting the performance of the elastic service. The performance of cloud computing applications which are running on Heterogeneous virtual machine cluster may be impacted. As a popular application of cloud computing, Hadoop-an open-source implementation of MapReduce is widely used for distributed processing model of large-scale data-intensive applications, and the configuration of parameters is crucial for the performance of MapReduce. As they expect that end users determine appropriate MapReduce parameters for running a job, which require in-depth knowledge of system and may lead to performance degradation, or use default configuration, especially in heterogeneous Hadoop circumstance.For a wide variety of virtual machine platforms, a heterogeneous virtual machine management platform is designed and implemented to achieve unified management and control for existing mainstream virtual machine management, and do not need to change the existing structure of the virtual machine management platform based on balanced allocation of virtual resources; also provide scalable adaptation layer interface and drive components, and support other heterogeneous virtual machine provisioning and management platform.For Cloud application of Hadoop on heterogeneous virtual machine platform, we propose a reinforcement learning approach to enable automated tunning configuration of MapReduce parameters. The RL approach has an initialization policy with offline learning to reduce online learn time in different circumstance, and through trial and error method iteratively update the Q value table to make configuration results close tc the optimal. Experimental results demonstrate that this approach car auto-configure the system, fast iterations to achieve convergence, have better computers performance and shorter running time.