Research On Active Suspension Control Strategy Based On Deep Reinforcement Learning Algorithm

Deep reinforcement learning can perform end-to-end self-learning of the complex high-dimensional mapping relationship between state and action on the basis that the model is completely unknown and there is no training set.At present,it has been applied successfully in the fields of game,robot control,chip design and intelligent driving.However,there is little research based on DRL in the field of automotive active suspension control,especially in aspects of active suspension vibration reduction and roll integrated control.Based on the theory of deep reinforcement learning algorithm,this paper proposes five reinforcement learning control strategies for the ride comfort and roll stability of active suspension system.The major research work is as follows:In view of the difficulty of obtaining the actual active suspension system dynamics model and the large amount of labeled sample data required by the neural network control,two model-free control strategies for active suspension are proposed which are based on deep Q network algorithm(DQN)and deep deterministic policy gradient algorithm(DDPG)respectively.The DQN strategy realizes the end-to-end mapping control of the system state to the actuating force by discretizing the actuating force and using the neural network to estimate Q value.The DDPG strategy uses neural networks to approximate the value function and the deterministic policy,and outputs the probability distribution of the action,which solves the problem of continuous actuating force control.The simulation results based on 1/4 active suspension show that the two reinforcement learning control strategies can effectively suppress the body vibration and DDPG strategy has better control performance.In order to solve the problems of low efficiency in the initial stage of reinforcement learning training and the possibility of violating the security constraints of the actual system by random exploration,an active suspension control strategy based on the safety deep deterministic strategy gradient algorithm(S-DDPG)is put forward by improving the DDPG algorithm.A half-car model of nonlinear active suspension with dynamic deflection and tire dynamic load safety constraints is modeled by Markov theory.Then,by introducing experience samples and pre-training,the exploration quality of algorithm training can be improved.Moreover,the ropout mechanism is used to improve generalization and fault tolerance.On this basis,considering the vehicle ride comfort under straight driving condition,the simulation is carried out in time domain and frequency domain.The results show that the proposed algorithm can effectively improve the stability of the suspension system and the stability of handling while improving the safety of the training process.In response to the demand for roll stability under high-speed steering,a roll control strategy of active suspension based on dual delayed deep deterministic policy gradient algorithm(TD3)is designed by directly controlling the roll moment and tracking the target roll angle as the main purpose.Specifically,a 6-dof roll-steering model is established,and the target roll angle is designed based on the reverse roll control mechanism.Then,the improved gradient reward function and the key concepts of the algorithm are designed.Finally,a sliding mode control strategy is designed for comparison.The simulation results confirm that the proposed strategy can quickly and effectively track the expected roll angle of different proportions,reduce the lateral load transfer rate(LTR),and improve the roll stability and roll limit.Aiming at the comprehensive optimization requirements of ride comfort and roll stability under straight-turn steering,a dual-mode fuzzy switching reinforcement learning control strategy(Fuzzy-TD3)integrating ride comfort control mode(RCM)and roll stability control mode(RSCM)is proposed by combining with fuzzy control and reinforcement learning algorithm.An integrated reward function with two modes is defined,and a dual-mode fuzzy switching control strategy with delay mechanism is designed to be used to decide the reward function of the TD3 algorithm.Futhermore an integrated control strategy with comprehensive control performance of ride comfort and roll stability is obtained through training.Finally,the numerical simulation under the excitation of the twisted road surface shows that the proposed strategy can comprehensively improve the vehicle’s smoothness and roll stability.