Research On Multi-Agent Deep Reinforcement Learning Method For Collaborative Collision Avoidance Of Ships

The maturity of big data,Internet of Things and artificial intelligence technology promotes the development of ships in the direction of intelligence,safety and green,and the problem of ship collision avoidance has always been the primary problem that must be solved in the intelligent navigation of ships.It is very common for many ships to meet in actual navigation.At this time,ships need to follow the International Regulations for Preventing Collisions at Sea(hereinafter referred to as "COLREGS Rules"),cooperate with each other,and jointly plan a reasonable collision avoidance strategy,so as to cope with the complex environment,achieve the goal of cooperative collision avoidance and reduce the risk of collision.Therefore,it is of great significance in both theoretical research and practical application to study the cooperative collision avoidance strategy of ships that meets the COLREGS Rules,which meets the needs of the intelligent development of ship navigation in the future.Using the perception and trial-and-error ability of deep reinforcement learning to learn collision avoidance strategies can not only ensure the safety of ship collision avoidance,but also ensure that the ship is more adaptable to the complex water environment.It has significant advantages in solving the problem of ship collision avoidance.However,most of the current researches on intelligent collision avoidance based on deep reinforcement learning ignore the interaction between ship collision avoidance,and there are problems such as lack of coordination in collision avoidance decision-making.Therefore,based on the multi-agent deep reinforcement learning method,thesis takes the construction of a multi-ship agent communication model for collaborative decision-making as the core,and conducts research on the multi-ship agent collaborative collision avoidance deep reinforcement learning method.The main work and achievements are as follows:(1)A multi-agent decision-making model for ship collision avoidance is constructedIn view of the fact that most of the current research only plans the collision avoidance strategy for own ship(the surrounding ships keep a fixed course and speed),and may not be able to deal with the uncertainty caused by the changes of the surrounding ships,the multi-agent deep reinforcement learning is used to study the collision avoidance method of the ship which ensures that multiple ships jointly plan collision avoidance strategies: a ship domain model and a ship collision risk assessment model is designed to reduce the collision risk;based on the multi-factor ship encounter situation identification model and the quantitative COLREGS Rules,a multi-ship collision avoidance strategy is designed to ensure its practicability;the similarity between the Markov Game process and the multi-ship collision avoidance process is studied,combined with the characteristics of ship navigation and collision avoidance,the state space,continuous action space and reward function are designed,the multiship agent collision avoidance decision model is constructed,and the multi-ship agent collision avoidance decision algorithm fused with rule constraints is proposed;experiments are designed to verify that the proposed algorithm can ensure the safety and practicability of ship collision avoidance.(2)A multi-ship agent communication model for collaborative decision-making is designedIn view of the current lack of communication among multi-ship agent,resulting in the lack of coordination of collision avoidance strategies,based on the agent communication method in multi-agent deep reinforcement learning,the cooperation mode of multi-ship agent is studied to promote the collaborative decision-making of ship agent: a multi-ship agent communication data extraction method based on attention reasoning is proposed,which quantifies the importance of the communication data from surrounding ship agent on itself,and extracts key data that is helpful for collision avoidance decision-making;a memory-driven multi-ship agent experience learning method is designed to learn its own navigation data and communication data from surrounding ship agents,accumulate experience,and promote ship agents to plan better collision avoidance strategies;combined with the proposed multi-ship agent decision-making model,a multi-ship agent communication model is constructed,and a collaborative decision-making method is designed by making full use of the ship’s own navigation data and the communication data from surrounding ship agents;comparison and simulation experiments are designed to verify that the proposed communication model can effectively ensure the coordination of multi-ship collision avoidance.(3)A deep reinforcement learning method for collaborative collision avoidance with multi-ship agent is proposedIn order to further enhance the safety and coordination of collision avoidance among multi-ship agent,a deep reinforcement learning method for collaborative collision avoidance of multi-ship agent is proposed: an exploration method for cooperative collision avoidance decision-making of ship agent based on noise network is designed to improve the probability of finding the best cooperative collision avoidance strategy;a ship agent evaluation network based on the multi-head attention mechanism is designed to promote the ship agent’s learning to be more conducive to its own acquisition of more rewarding information,thereby enhancing the synergy of the ship agent’s learning;based on the multi-ship agent communication model,combined with the above research,a multi-ship agent collaborative collision avoidance decisionmaking model is constructed,and COLREGS Rules are introduced to propose a multiship agent collaborative collision avoidance deep reinforcement learning algorithm;comparison and simulation experiments are designed to show that the proposed method further improves the coordination and safety of ship collision avoidance in the multiship encounter situation,and ensures the practicability of the collision avoidance strategy.