A Model Training Service System For Data Sharing

In the era of the Internet of Everything,massive amounts of data are generated every day.Data is the booster of artificial intelligence development,and artificial intelligence applications based on deep learning are inseparable from the support of highquality and large-scale data.However,in reality,there are very few publicly available data sets,and a large amount of data is in a closed and isolated state.How to eliminate data silos and promote the safe circulation and sharing and utilization of data is an urgent problem to be solved at present.The current development of data sharing and trading platforms is still in its infancy.The products provided are mainly basic data and data package download services.After payment,buyers have access to the data permanently or for a specified period,which is prone to sensitive information leakage and illegal data copying dissemination and other issues.Providing users with a data functional service interface rather than the data itself can solve the contradiction between data protection and data use to some extent.Aiming at the demand for training data of deep learning models,this paper designs and implements a model training service system based on the institution’s own data sets and its own computing resources,which can automatically complete the model training tasks according to the user-defined model structure and specified data sets and return them to the user,so that the data can be used and invisible.The main contribution of this paper is to propose a resource allocation and scheduling strategy suitable for data sharing scenarios.On a small computer cluster,a model training service prototype system for data sharing is implemented,which completes the whole process from receiving user requests to automatically allocating resources,deploying jobs,and returning training results,and provides users with a convenient and easy-to-use interface.This paper involves three groups of experiments.The first group evaluates the performance of different systems.The experiments show that the system has the advantages of fast response to service requests and short work completion time under the workload combination of the next system.The resource allocation and scheduling strategies are changed under different work efficicncies.The experiments show that the different strategies proposed in this paper have better performance.The system uniformly allocates training services in data scenarios that are more suitable than user-specified resources.