| With the rapid development of leading technologies such as artificial intelligence and 5G,as well as the urgent demands of human beings on the safety of driving vehicles and the alleviation of traffic congestion,intelligent automobiles have become the focus of the modern automobile industry development.The combination of artificial intelligence algorithm and autonomous driving technology has become the research hotspot of scholars all over the world.As an outstanding algorithm with decision-making function in the field of artificial intelligence,Deep Reinforcement Learning(DRL)has many advantages such as autonomous learning in solving complex control tasks.And it is of great significance to realize intelligent control of driverless vehicles.In this paper,the depth reinforcement learning algorithm is applied to the learning of intelligent control strategy of autonomous driving,and the effectiveness of the algorithm is verified by The Open Racing Car Simulator(TORCS).In order to solve the problem of overvaluation deviation and weak training parameter in Deep Deterministic Policy Gradient(DDPG)algorithm,an improved Twin Delayed Deep Deterministic policy gradient(TD3)algorithm based on DDPG algorithm is proposed as an intelligent control algorithm for driverless vehicles.Secondly,to solve the problem that the dynamic of the value estimation is not taken into account because of the policy updating with fixed delay step size in the TD3 algorithm,so that the learning efficiency of the policy is affected.In this paper,a Twin Delayed Deep Deterministic Policy Gradients with Dynamic Delayed policy update(DD-TD3)is proposed.DD-TD3 sets the exponential weighted moving average of the update loss of value network as the update standard of dynamic delay step,and guides the update of policy network delay based on this standard.Then,the experience samples in the replay buffer are sampled randomly in the training process for the TD3 algorithm,and the importance difference of the experience samples is not considered,which leads to the problem that the low training efficiency of the algorithm and the unstable training process.In this paper,an Emphasizing Outstanding Experience Buffer(EOE Buffer)is designed,which classifies the importance of experience samples by means of the EWMA of immediate rewards.The training efficiency of the algorithm is improved by increasing the sampling frequency in the outstanding experience samples.For the problem of the catastrophic forgetting problem in neural network,through deeply studying the importance of the initial exploration experience in the training process of deep reinforcement learning algorithm,the initial exploration experience is reserved in EOE Buffer to improve the continuous learning ability of TD3 algorithm.Experimental tests were carried out in the Pendulum-v0 environment,the validity of DD-TD3 algorithm and EOE is proved.Finally,the DD-td3 algorithm based on EOE Buffer(EOE DD-TD3)is used to control the intelligent driving of the vehicle in the TORCS simulation environment.Through the analysis of the experimental results,the effectiveness of the algorithm in learning the intelligent control strategy of unmanned vehicle is verified. |
