Design Of Formation Control Algorithm For Autonomous Underwater Vehicles Based On Deep Reinforcement Learning

In recent years,with the development of marine engineering technology,autonomous underwater vehicle(AUV)formation control has been widely used in military and civil applications such as intrusion surveillance,undersea terrain detection,underwater track planning,and seawater detection.With the deepening of human development and research in the ocean,multiple autonomous underwater vehicles work together in formation with higher efficiency and fault tolerance when facing complex ocean tasks.Formation control of autonomous underwater vehicles is essential to improve the stability of autonomous underwater vehicles working together.However,due to the complexity of the underwater environment,it is difficult to accurately solve the model parameters of underwater vehicles,and the attenuation of underwater acoustic channels is severe,which limits the communication of underwater vehicles.Based on this,this paper conducts a study on model-free formation control based on deep reinforcement learning,considering the impact of uncertain model parameters,and unknown underwater channels on formation control.The specific work is as follows:(1)For the uncertain model parameters of an underwater vehicle,a deep Q-learning(DQN)algorithm is designed for the formation controller to get rid of the dependence on the model.Consider the formation model of leader-follower to realize the track planning in the formation control process and the mission of formation maintenance.To solve the problem that the parameters of the underwater vehicle model are uncertain and it is difficult to obtain the underwater vehicle model,a deep reinforcement learning algorithm is proposed to learn the underwater vehicle control strategy by interacting with the environment autonomously,so as to get rid of the dependence of the formation controller on the underwater vehicle model.Based on this,a DQN-based queue controller is designed.To solve the problem of uneven track and slow convergence of DQN algorithm,an improved DQN queue controller is presented.An adaptive DQN queue control algorithm based on feedback error action is presented.The output of DQN action strategy is adjusted based on feedback action error,which improves the smoothness and convergence of queue control.Finally,the queuing effect of the two algorithms is verified by simulation.(2)An optimization framework for formation control under communication constraints is proposed for the effect of communication constraints on formation control of autonomous underwater vehicles.A submarine underwater vehicle formation control system is considered,and based on the submarine data transmitter,a least square estimator based on environmental sampling data is proposed to predict the unknown channel parameters in fading environment.The SNR is calculated according to the predicted parameters of the channel estimator,and the joint optimization problem of communication constraints and formation control is constructed.Considering that the underwater vehicle is a continuous control quantity in the actual control,the dynamic model of the underwater vehicle with four degrees of freedom is adopted as the research object,and based on this,a formation control framework of depth deterministic strategy gradient formation control algorithm optimization communication constraints is proposed.Finally,the effectiveness of the formation control algorithm is verified by simulation and experiment.