Research On Reinforcement Learning Control Method For Intelligent Underwater Vehicle Movement

Autonomous Underwater Vehicles(AUV)is an important marine technology equipment.With the increasing demand for marine development in recent years,more and more scholars and research institutions have begun to pay attention to AUVs.However,complex tasks and changing marine environments require AUVs to have certain self-learning capabilities to sense and evaluate the surrounding environment,which makes autonomous decisions for intelligent control.In order to introduce the self-learning ability into the control tasks of AUVs so that the adjustment of parameters which relies on the accumulation of artificial experience in traditional control methods can be avoided,a series of research work is carried out around the reinforcement learning control algorithm of AUVs motion.Combining deep learning and strategy gradient methods,a reinforcement learning control method for AUVs which output continuous actions is designed.Among them,in order to make better use of the motion data of the AUVs,a priority traversal idea is introduced to give priority to each piece of data;in order to solve the problem of the slow convergence speed of the reinforcement learning algorithm caused by the complex kinematic characteristics of the AUVs,a greedy strategy parameter calculation method is designed to make the parameter value gradually decrease with the training time to improve the convergence speed.Markov modeling was performed for different control tasks of the AUVs.Using the above reinforcement learning controller,the rudder-wing AUV and the vertical push AUV were simulated under multiple operating conditions and the results were analyzed.For the depth control task of AUVs,the influence of different reward function coefficients on the control effect was studied.Aiming at the characteristics of low data utilization rate and volatile control effect when the reinforcement learning algorithm is used in the control of AUVs,a distributed reinforcement learning control method for AUVs is designed through distributed parameter update and the use of parallel structures.Finally,rudder-wing and vertical push AUVs were used to carry out multi-mode simulation comparison experiments to verify their superiorityTo the best of our knowledge,this is the first time the research on reinforcement learning control algorithms for AUVs has been carried out for many complete physical experiments in China.Carefully studied the problems involved in physical experiments:communication schemes,Markov modelling,and how to design the reinforcement learning algorithms applied to real AUVs are carefully studied.Finally,the test results and existing problems and reasons are analyzed.