Research On Neural Network Training Algorithms For Artificial Intelligent Chips In Edge Computing

Edge computing is an important means to effectively address issues such as privacy protection,bandwidth constraints,and communication latency,and is also a key technology for achieving the Internet of Things.In the application scenarios of edge computing,artificial neural network algorithms are an important method for implementing intelligent decision-making at the edge.With current strict limitations on the size and power consumption of artificial intelligence(AI)chips and highly dynamic edge computing application scenarios,the efficient and low-cost training of artificial neural network model parameters has attracted high attention from governments,large enterprises,and research institutions around the world.To address the above issues,current research mainly focuses on cloud-based training and edge-based deployment to implement neural network inference at the edge.However,in highly dynamic edge computing application scenarios,it is necessary not only to complete pre-training of the neural network model in the cloud to obtain a higher accuracy initial model,but also to achieve online training of the model during the inference process on the edge AI chip,in order to continuously improve the model accuracy.On the one hand,the pre-training task completed in the cloud is less affected by hardware computing power limitations.Therefore,to obtain a high-precision initial model,it is important to address the significant issues of slow convergence speed and weak generalization ability of training algorithms.On the other hand,the online training task performed at the edge is severely affected by the size and power consumption of the edge AI chip.Therefore,to achieve continuous improvement of model accuracy in edge computing scenarios,it is essential to solve the key problem of high computational cost associated with training algorithms.To address these significant challenges,this paper proposes systematic training algorithms for fast convergence,high generalization,and low computational cost,respectively,in both the neural network pre-training phase and the edge-based online training phase.These solutions provide feasible approaches to achieving sustainable improvement in the accuracy of neural network models on AI chips.The main research contents of this paper are as follows:1.Fast-Converging and Generalizable Training Algorithms for the Pre-Training StageAn Adaptive online optimization algorithm under strong convexity.To address the problem of slow convergence in convex optimization algorithms,this paper proposes a fast-converging adaptive online optimization algorithm.First,the form of the second-order moment in the adaptive algorithm is redesigned,and it is analyzed that the step size selection of the algorithm is closer to the ideal step size under this form.Second,it is theoretically proven that the proposed algorithm can achieve a regret bound of O(log T)in the online theoretical framework,where T represents the number of iterations,which is faster than the convergence rate of traditional adaptive convex optimization algorithms(O((?))).Finally,comprehensive simulation experiments are conducted on publicly available datasets,and the results show that the proposed algorithm has faster convergence speed and better generalization ability.A Fast Adaptive Optimization Algorithm Based on Bandit Sampling.To address the interference problems of invalid samples to model training,this paper proposes a fast adaptive optimization algorithm based on Bandit Sampling,which selects effective training samples in each iteration using bandit sampling method.This not only improves the convergence speed of the algorithm,but also enhances its generalization ability.When the loss function is convex,we propose a Bandit Sampling-based adaptive online optimization algorithm,AdaBeliefBS,and prove its boundedness with a determined upper bound.Specifically,when the feature vectors follow a double-tailed distribution,the regret bound of AdaBeliefBS is O(d(?))+O((?)),which is tighter than the regret bound of the original optimization algorithm.When the loss function is strongly convex,we propose a Bandit Sampling-based strongly convex optimization algorithm,SAdamBS,and prove its convergence.When the feature vectors follow a double-tailed distribution,the regret bound of SAdamBS can be tighter and is O(d log(Tlog2N/KN2logd)).Simulation results demonstrate that the proposed two improved algorithms converge faster and have stronger generalization ability than their original versions.2.Low Computational Cost Training Algorithms for online training on edge AI ChipsA Projection-Free Adaptive Online Convex Optimization Algorithm.To address the problem of calculating high order projection operators of optimization algorithms,this paper proposes a new low-computational-cost adaptive training algorithm called LightAdam,which replaces high-order projection operators with onedimensional linear search steps using Frank-Wolfe optimization technique.The proposed algorithm greatly reduces the iteration cost and enables online training of deep models on edge AI chips.The regret bound of the algorithm is proved to be O(T3/4)and this theoretical result is verified by multiple sets of simulation experiments.To maintain the algorithm’s generalization ability,the momentum form is redesigned and its generalization performance is improved and validated through both theoretical and simulation experiments.Under convex conditions,LightAdam’s computational cost is more than 15%lower than that of other mainstream algorithms.The computational cost of the proposed algorithm completely satisfies the power consumption requirements of edge AI chips.A Random Coordinate Block Adaptive Optimization Algorithm for NonConvex Conditions.To address the problem of high cost of high dimensional eigenvector gradient calculation,this paper proposes a randomized coordinate block adaptive optimization algorithm called RAda.This algorithm utilizes the technique of randomized coordinate blocks to randomly select a coordinate block of high-dimensional feature vectors for gradient computation during iteration,thus solving the problem of high computational cost of high-dimensional feature vector operations,and providing another option for online training on edge AI chips.Theoretical proof shows that the algorithm converges under non-convex conditions,and the first-order stochastic complexity of the algorithm is O(1/δ2),achieving a solution accuracy of δ.Simulation experiments show that the computational cost of the proposed training algorithm under non-convex conditions is reduced by more than 24%compared to other mainstream non-convex training algorithms.RAda provides an effective solution for implementing online model training on edge devices with strict power constraints.In summary,this paper addresses the challenging issues of limited computational resources,power consumption,and cost resources in edge AI chips.Through model pre-training and online training,as well as chip-side simulation experiments,efficient and low-cost neural network training algorithms are studied for edge AI chips.This research has important theoretical significance and practical value.