Research On Formation Obstacle Avoidance Mechanism Based On Reinforcement Learning In Unknown Environment

In recent years,with the rapid development of artificial intelligence,autonomous driving,and sensor technology,the functions of modern unmanned ground vehicles(UGVs)have become increasingly rich.Due to their advantages of small targets and high flexibility,they are widely used in civilian and military fields.The UGVs cluster system is a complex system composed of multiple UGVs,which have higher efficiency,robustness,and reliability than a single UGV.When the UGVs cluster system performs tasks in unknown environments,it needs to maintain a specific formation while avoiding obstacles safely and efficiently.Moreover,when performing special tasks such as reconnaissance and attack,the unmanned vehicle cluster needs to be able to switch to the designated formation according to the commands issued by the terminal.This paper takes the UGVs cluster as the research object,aiming to enable it to have the ability of formation keeping,formation obstacle avoidance,and formation switching in unknown environments.The main research content and innovation of this paper include the following aspects:(1)Traditional leader-follower methods can calculate virtual target points that are prone to conflict with obstacles in environments with static or dynamic obstacles,resulting in the phenomenon of UGVs oscillating near obstacles.A Virtual Obstacle Avoidance Point(VOAP)algorithm is proposed to calculate a substitute target point in the pre-forward direction of the UGVs cluster,improving the smoothness of the cluster’s motion trajectory and the efficiency of the cluster’s formation recovery after obstacle avoidance.Secondly,considering that most of the obstacle avoidance algorithms currently used in cluster formation systems scan and preprocess known maps before performing path planning,they lack the ability to explore and avoid obstacles in unknown maps.To address this issue,this paper proposes an adaptive formation obstacle avoidance algorithm for the UGVs cluster based on the Twin Delayed Deep Deterministic Policy Gradient(TD3)algorithm,comparing with the traditional cluster obstacle avoidance algorithm,the collision rate and the total path of the cluster obstacle avoidance in unknown environment are reduced.Moreover,the reward function is improved based on the idea of the Artificial Potential Field(APF)method,adding a warning distance around obstacles and calculating different levels of training rewards based on the distance between UGVs and obstacles to improve the safety of formation obstacle avoidance.(2)In response to the need for passive or active formation switching in practical applications of the UGVs cluster system,this paper studies formation switching algorithms based on the UGVs cluster formation obstacle avoidance algorithm.First,considering two common formation switching scenarios: Efficiency-First Formation Change(EFFC)and Power-First Formation Change(PFFC),mathematical models are established for each,and differences with the traditional assignment model are analyzed.Secondly,considering that the Hungarian algorithm,used to solve the traditional assignment model,only addresses the problem of minimizing the total moving distance and is not suitable for the above two formation switching scenarios,and accuracy issues arise when solving high-dimensional models.Therefore,a matrix transformation based formation switching algorithm for the UGVs cluster was proposed to solve the maximum efficiency and endurance problems in formation switching scenarios.The correctness and efficiency of this algorithm are verified in two formation-switching simulation scenarios.(3)This paper presents a control system for the UGVs cluster that integrates a reinforcement learning-based formation obstacle avoidance algorithm and a matrix transformation-based formation switching algorithm.The system incorporates the Joint Architecture for Unmanned Systems(JAUS)toolkit,which provides discovery services and connection management services for each UGV in the cluster.The functions of formation establishment,formation movement,formation obstacle avoidance and formation switching of the UGVs cluster are provided for users,and the research content of this paper is applied and realized.