Research And Implementation Of Key Technologies Of Federated Learning Foundation Platform

Data is one of the three elements of artificial intelligence.In the era of big data,with the increasing emphasis on data security in various industries,and the promulgation of relevant laws that further restrict data trading and circulation,data owners are unwilling to share data,forming their own data islands over time,which in turn affects the development of artificial intelligence.The emergence of federated learning technology can effectively solve the problem of data silos.It can utilize dispersed data from various physical locations to collaboratively train a model while ensuring data security.The difference in accuracy between the model and the model trained through non federated learning is within an acceptable range.Today,federated learning has been well applied in fields such as healthcare and finance.With the development of technology,research on federated learning technology remains a hot topic in various fields.This thesis takes federated learning technology as the core,combines Kubernetes’ containerized choreography management technology,privacy protection technology and edge computing ideas,and designs and implements the key technologies in the basic platform of federated learning:(1)Designed and implemented a task scheduling engine for task splitting,scheduling,and operation management in the basic platform.Responsible for splitting executable algorithm tasks from the task flow,scheduling them to the correct data nodes,and using Kubernetes Jobs to execute algorithm tasks.The task scheduling engine is also responsible for monitoring the status changes of tasks and algorithm tasks,to avoid some completed tasks being continuously restarted due to status errors.In addition,the scheduling engine also supports suspending running tasks and restoring suspended tasks to their runtime state and continuing to run.(2)Designed and implemented an algorithm execution engine that provides environment and capability support for algorithm operation.Using the LMDB database to complete data storage,a data object for algorithm tasks was designed,and a series of data access and calculation methods were provided to meet the calculation requirements of the data object.And a complete multi-party communication process has been designed to support all participants in completing collaborative training.In addition,the basic platform also supports developers to create new algorithms and have them executed by the algorithm execution engine.Finally,this thesis uses the same batch of data,the same model,and the same model structure to train separately through federated learning modeling and nonfederated learning modeling.The model results of the two training methods are compared,and in the federated learning modeling method,the testing of the two key technologies mentioned above is completed.The final results indicate that the functions of the algorithm execution engine and task scheduling engine are consistent as expected,and the accuracy of the two model results is not significantly different.The model obtained through federated learning modeling is still usable.