Research On Rhythmic Motion Control Method Of Quadruped Robot Based On Deep Reinforcement Learning

Quadrupeds have not only flexible athletic ability but also strong power to traverse challenging terrains.For example,cheetahs can run at a speed of more than 100 kilometers per hour,and ibexes can climb cliffs.Compared with wheeled or tracked robots,bionic quadruped robots have better dynamic movement capabilities in complex filed environments.Especially in applications like searching and rescuing in disaster areas,outdoor patrol,and industrial park security,quadruped robots have irreplaceable value.The traditional modeling control methods for quadruped robots,such as central pattern generators,model predictive control,whole-body control have achieved good results.However,due to the high nonlinearity of the robot model,these methods require accurate dynamic models and complex robot theoretical foundations and tedious gait design process.In recent years,the development of deep reinforcement learning provides an idea for automatically designing motion control policies,which greatly simplifies the design process.However,reinforcement learning requires that reward functions can adequately describe the quadrupedal movement,and designing such rewards is often difficult because small changes in rewards can lead to large differences in robot behaviors.In addition,the training process of reinforcement learning is a bit slow,which makes the designers have to spend a lot of time and effort adjusting reward functions to acquire desired rhythmic locomotion behaviors.Inspired by the biological rhythmic locomotion mechanism,this paper proposes a rhythmic motion control method for quadruped robots based on deep reinforcement learning,including two major innovations:rhythmic motion control architecture and action space of joint position increments.The rhythmic motion control architecture simulates central pattern generators in the nervous system of the spinal cord,which consists of a rhythm generator network and a pattern formation network.The former generates the rhythmic motion signal of the robot,and the latter outputs the corresponding joint position commands.This control architecture directly embeds the rhythmic locomotion pattern into the policies,rather than being completely specified by reward functions,reducing the difficulty of adjusting reward functions.The action space of joint position increments restricts the output of the reinforcement learning policies so that the joint position commands are limited to the vicinity of the current joint positions,which can avoid excessive torque output of the joint motors and narrow the action range of the policies to speed up the training of reinforcement learning.On the basis of this method,a teacher-student model is introduced to extract the surrounding environmental features from a sequence of proprioceptive states,which further enhances the ability of blind quadrupedal locomotion under complex terrains.This paper fully validates the proposed rhythmic motion control method on simulated and real-world quadruped robot platforms.Comparative experiments with previous work demonstrate that the proposed method can naturally stimulate rhythmic locomotion behaviors while reducing the difficulty of adjusting reward functions and speeding up the training of reinforcement learning.In the real-world deployment experiments,the quadruped robot shows the abilities of full-spectrum movement and impact resistance,and successfully traverses a variety of challenging terrains,such as stairs,ice,snow,mountains and slopes,demonstrating the effectiveness and robustness of our method.