| In recent years,Autonomous Underwater Vehicles(AUVs)have undergone rapid development,and AUV path tracking control has become one of the key research directions in the field of underwater robot control.Traditional AUV path tracking control involves establishing a dynamic model and solving for the optimal control quantity for path tracking.However,the complexity of the underwater environment makes it difficult to establish an accurate dynamic model,and the path tracking accuracy is greatly constrained by the established model.Some research has applied Reinforcement Learning(RL)to AUV path tracking,which does not require a dynamic model and instead learns expected behavior through a large amount of interaction with the environment,greatly reducing the workload of controller design.However,the RL controller obtained by training in a single ocean environment,and path tracking of AUVs in dynamic ocean current disturbances presents a significant challenge,in which RL-based controllers may perform poorly or even fail.MetaReinforcement Learning(Meta-RL)is based on RL and can adapt to new testing tasks through a small amount of training tasks,offering a solution to the path tracking problem under ocean current disturbances.Therefore,this paper proposes a method based on MetaRL to effectively perform path tracking under ocean current disturbances.The main research work of this paper includes:(1)Build an AUV path tracking simulation environment.Addressing the shortcomings of existing AUV simulation environments in the RL training field,this paper proposes a new simulation environment based on the Unreal Engine(UE)that features highly realistic physical simulation and can simulate the interference of ocean currents on the AUV.Additionally,the state space,action space,and reward function are uniformly designed for RL training,supporting million-step RL and Meta-RL training and resolving the problem of high time cost associated with large-scale RL training in existing AUV simulation environments.(2)Propose a Meta-RL-based path tracking control method under ocean current interference.To address the impact of ocean current interference on the controller’s performance in AUV path tracking,the paper uses the PEARL method based on task inference in Meta-RL to solve this problem.Firstly,the paper decomposes ocean current interference into multiple different fixed ocean current interferences,and then conducts RL learning and decision-making under these fixed ocean current interferences.When the ocean current interference changes,the controller adapts to the new ocean current interference by inferring the task inference information from the inference network.The paper proposes an improved PEARL-SA(PEARL with State-quantized and Attention),which designs a quantization module to discretize the state of the AUV,constraining the infinite continuous state space dimension to the size of the quantization table,to solve the problem of the AUV state space being infinitely large and possibly noisy.It also designs an inference network with Self-attention to infer ocean current interference information from observable AUV states,helping the RL-based controller make better decisions.(3)Verify the effectiveness of the PEARL-SA control.The path-following control experiment is carried out in the constructed simulation environment,and the path-following performance is compared with the dynamics-based and RL-based controllers under different ocean disturbances,and the robustness of PEARL-SA under the influence of noise is evaluated.The experimental results show that under the interference of ocean currents,PEARL-SA is more effective than dynamics-based and RL-based controllers in controlling AUVs for path tracking,and is more robust than RL-based controllers,and can still be effective in noisy environments Do path tracing. |
