Research On Multi-Task Reinforcement Learning Based On Parallel Training

With the development of reinforcement learning,its combination with deep neural networks has been widely used in many fields.However,current reinforcement learning agents are often only trained for specific tasks and lack the ability to handle multiple tasks.This means that a specific agent needs to be trained for each task,resulting in huge time and space costs.Therefore,improving the multi-task processing ability of reinforcement learning becomes particularly important.Existing multi-task reinforcement learning methods have problems such as insufficient feature extraction and uneven training progress between tasks.In addition,most of these methods adopt serial training methods,which do not fully utilize the performance of current multi-core processors,further increasing the cost of training.To solve the above problems,this paper proposes the following three parts of research content based on parallel training:(1)Parallel multi-task reinforcement learning algorithm based on feature separation.Existing methods for feature extraction focus on the individuality of tasks,without fully considering their commonalities.As the foundation of a multi-task environment,the commonalities among tasks can effectively promote positive transfer between tasks,thereby improving the speed and effectiveness of model training.To address this issue,a parallel multi-task reinforcement learning algorithm based on feature separation is proposed.This algorithm trains a shared feature extractor and individual feature extractors simultaneously to extract common and individual features,respectively.These features are then concatenated and used for decision-making by the agent.Experimental results show that the proposed algorithm can effectively extract common and individual features among tasks to assist the agent’s decision-making and improve training effectiveness.(2)Parallel multi-task reinforcement learning algorithm based on task similarity clustering.Current methods are mostly based on the assumption that tasks in multi-task environments are strongly related to each other.However,in practical application environments,there may be no connection between tasks or even opposite goals between them,that is,there are fewer commonalities between tasks.To solve this problem,a parallel multi-task reinforcement learning algorithm based on task similarity clustering is proposed.The algorithm defines the similarity between tasks under multiple tasks and calculates it with gradient information.It expands group training according to similarity and improves the commonality between tasks within groups and positive transfer during training.Experiments show that parallel multi-task reinforcement learning methods based on task clustering can obtain better task clustering results under controllable complexity,thereby improving the effect of multi-task learning.(3)Parallel multi-task reinforcement learning algorithm based on progress balance.There is often an imbalance in training progress between multiple tasks,resulting in poor training results for some tasks.To solve this problem,a parallel multi-task reinforcement learning algorithm based on progress balance is proposed.The algorithm considers the training progress of each task and determines the number of trajectories that should be extracted for each task in the current batch according to the training progress of each task in calculating and backpropagating losses in the intelligent agent model.At the same time,output entropyrelated hyperparameters in the algorithm are used as weight coefficients for loss calculation to further adjust the training progress of each task in calculating final losses.Experimental results show that progress balance algorithms can effectively balance the training progress of tasks in a multi-task environment and improve the performance of tasks with slow training progress.For the methods proposed in this research content,this paper has conducted a large number of experiments and analyses and verified the effectiveness and superiority of this paper’s methods through comparison with other methods.