Design And Implementation Of Big Data Experiment Cloud Platform Based On OpenStack

Faced with problems encountered in the teaching process of big data experiment courses,we propose to provide a professional platform for a centralized management of big data experiments.Therefore,we carry out a design and development of a cloud platform for the management of the experiments of big data courses by utilizing an open source cloud platform,OpenStack,which is based on virtualization technology.According to the teaching requirements,we equip the platform with a management module of experiment clusters,a teaching management module and a platform monitoring module,and integrate multiple physical servers through OpenStack to provide users with virtualization services.In the management module of experiment clusters,one-click rapid deployment of cluster is realized by constructing a cluster pool and using "pre-deployment real allocation" method,and the function interface of OpenStack is used in allocating,saving and recovering experiment environments.The import of the toolsets from the server side to the experiment cluster is completed through the remote transmission technology.In the teaching management module,under the OpenStack's right management,we design and implement user isolation and quota restrictions for the teaching system.Automating operations such as creation and binding of OpenStack projects and users,are used to achieve the benefits of batch processing users.The entire processes of managing experiment teaching,including experiment release,information management,job uploading and online review,are implemented and integrated into the Horizon subproject.In the platform monitoring module,the monitoring pressure of the OpenStack controller node is shared by multi-node physical machine,which queries the physical resource information for itself by a script automatically deploying on it.And the monitoring service is written as a system daemon,so that it can be booted.A virtual machine information collector and a user experiment behavior collector are implemented to collect relevant information.The platform is based on the browser/server architecture.The specific implementation is based on Python and scripting programming language.And we exploit Django framework in MVT mode to achieve the secondary development of OpenStack,and integrate it into the B/S system of OpenStack.We carry out tests on the platform with several examples of typical big data experiments.The results show that the platform can prominently shorten the time of creating big data experiment environments,and can monitor resources occupancy at user level in real time.