Path Following Of Unmanned Surface Vehicles Based On Reinforcement Learning

Reinforcement learning is developing rapidly in recent years,accumulated a lot of achievements,and it has been widely applied to related problems in various fields.It has been applied in unmanned surface vehicles to solve key technical problems such as environmental perception,condition monitoring and fault diagnosis,path planning,energy efficiency control,and autonomous navigation,so as to meet the growing market demand for intelligent ships for energy conservation,environmental protection and cost reduction.As one of the key technologies of unmanned surface vehicles,path following control is essential for a safer,more economical and more efficient intelligent ships.The design of path following controller with good dynamic responsiveness,high control accuracy and great capacity of resisting disturbance is the basis of realizing more energy saving and high efficiency of unmanned surface vehicles.This paper combines the adaptive line-of-sight navigation algorithm and the depth deterministic strategy gradient algorithm in reinforcement learning,proposes a model-independent adaptive PID controller and adaptive linear active disturbance rejection controller,and realizes the path following control of unmanned surface vehicles.Main research contents and results are as follows:（1）This paper analyzes and summarizes several main methods of path following control from navigation,control and perception.（2）Taking the scale ship model as the basis,the unmanned surface vehicles simulation test platform is built.The least square method was used to identify the parameters of the quadratic response ship model and the validity of identification parameters is verified by comparing the simulation data with the actual voyage data.（3）In order to realize the course tracking control of unmanned surface vehicles,deep deterministic policy gradient is used to adjust the parameters of PID controller,which can improve the dynamic responsiveness and the capacity of resisting disturbance of PID controller.In addition,based on the linear active disturbance rejection control theory,the linear extend state observer is used to estimate the uncertainties and environmental disturbances,which is compensate in the output of the controller based on deep deterministic policy gradient.Compared with the traditional PID controller,the simulation results show that the adaptive controller based on deep deterministic policy gradient has better dynamic responsiveness and greater capacity of resisting disturbance,among which the adaptive linear active disturbance rejection controller has the greatest capacity of resisting disturbance and the steering command is smoother.（4）Based on particle swarm optimization,an adaptive line-of-sight navigation algorithm is designed,which transforms the path following control into the course tracking control.Then combine the adaptive line-of-sight navigation and the above two kinds of course tracking controllers,the path following control of unmanned surface vehicles is realized.Through the simulation comparison,it is proved that the two adaptive controllers designed in this paper have higher tracking control precision.