Collision Avoidance Navigation And Control For Unmanned Marine Vessels Based On Reinforcement Learning

To realize unmanned marine vessels with highly autonomous control is the new goal of shipbuilding and shipping industry.In order to realize autonomous navigation of unmanned marine vessels(UMV),autonomous navigation is the core of navigation safety,and automatic motion control is the key of manoeuvring a vessel to the port of destination.For above two key problems,a suit of collision avoidance navigation and motion control method applied to engineering application is proposed and validated for UMV in this thesis,which is based on the theory of autonomous reinforcement learning and fully complies with the Convention on the International Regulations for Preventing Collisions at Sea(COLREGs)and navigation experience together with the manoeuvring characteristic of vessels.A novel intelligent collision avoidance algorithm based on deep reinforcement learning(DRL),is proposed for the difficult problem of multi-vessel at complex navi-gational conditions.By incorporating the manoeuvrability of vessels into the design of the reinforcement learning task for avoiding collision,which reduces the time of train-ing model and promotes the applicability of the collision avoidance model.Because the collision avoidance decisions are made by considering the own vessel's whole naviga-ble area obtained by transforming the navigation experience and rules into the dynamic navigational limitation polygons or lines,the obtained decisions not only are in accord-ance with the navigation experience and rules,but also can adapt to different complex environments by adjusting experience parameters.Finally,a series of comparison tests of the numerical simulations and the collision avoidance experiments are conducted by three self-propelled vessels with different scale and maneuverability,by using the same collision avoidance model of reinforcement learning.The results are highly consistent and demonstrate that the collision avoidance algorithm has an effective ability of obsta-cle avoidance.It is great helpful to solve the problem of automatic collision avoidance with complex multi-vessel navigational condition for UMV at open sea and in restricted waters.In order to further improve the ability and efficiency of obstacle avoidance of UMV,two multi-layered collision avoidance navigation methods based on the COLREGs and navigation experience are proposed,which have complied to the re-quirements of "early,large,wide,and clear" for the give-way vessel to avoid the danger of collision and the close quarters situation.Based on the A*algorithm combining with the manoeuvring characteristics of vessels,and fully considering the navigation experi-ence and rules,two approaches of collision avoidance navigation with parallel deci-sion-making for only changing the course or changing the course and speed simultane-ously are proposed.In addition,a collision avoidance control strategy is established by the vessel safety Bumper model.Furthermore the two approaches of collision avoidance navigation are fused together with the proposed DRL-based collision avoidance algo-rithm respectively.As a result,two methods of multi-layered collision avoidance navi-gation for only changing the course or changing the course and speed simultaneously are obtained.Finally,the simulations for above two methods are conducted and the results demonstrate the effectiveness of these methods.Aiming at the control problem of course and path-following of UMV,an approach of course control using actor-critic algorithm,based on the idea of autonomous rein-forcement learning,is proposed firstly.In order to realize the interference compensation of UMV's rudder angle,the disturbance observer is adopted to estimate the disturbance accurately.Thus,a course control approach with the disturbance observer based on re-inforcement learning is proposed.Furthermore,a path-following control approach based on reinforcement learning is also proposed by applying the idea of indirect path-following control.Finally,the simulations including the disturbances of wind,waves and ocean currents,are conducted to validate the control algorithms of course and path-following and the results show that the three algorithms have a favoable con-trol responses,and their manoeuvring behaviors are in accordance with good seaman-ship.In order to facilitate the rapid debugging and validation of the autopilot algorithm during the process of engineering practice,and to relieve the reliance of autopilot tests on sea trials,which features high risk,long debugging period and high expenses,a sim-ulation system for testing marine autopilot control algorithm is developed,which is based on the international standard electronic chart and similar to the sea trial conditions.At last,an example of path-following autopilot test is demonstrated to validate the ef-fectiveness of this system,which shows that the system is useful for promoting the re-search and application of autopilot control algorithms.In order to realize autonomous and safe navigation of UMV,a new path-following controller with collision avoidance navigation based on reinforcement learning is de-signed and obtained by combing the collision avoidance navigation method with the path-following control approach.The simulation system for testing marine autopilot control algorithm is also improved to establish the test platform of the new path-following controller.Finally,the simulation results of UMVs' path-following con-trol under the disturbances with wind and currents demonstrate that the new path-following controller not only makes the UMVs navigate well along the desired route,but also takes effective evasive manoeuvres in accordance with the COLREGs,which ensures the navigation safety of UMVs.