Researches On Adaptive Critic Learning Control Approaches For Intelligent Driving Vehicles

As the core of intelligent transport system,the intelligent driving technologies of autonomous land vehicles(ALVs)has broad application prospects,which has been a new focus of the world's automotive industry competition.Motion control is the basic technology to realize intelligent driving.However,the challenge in theory and technology still exists to control ALVs with high performance due to the complex dynamic model and the changing road environment.As an important research direction in reinforcement learning(RL),the adaptive critical learning(ACL)control methods has good self-learning ability,independent characteristics of model information and data-driven features.Therefore,ACL control methods are commonly used for solving the control problems in the domains of robotics and complex control systems.Confronted with the difficulties,such as the complex driving environment,the nonlinear dynamic model with the time-delay characteristics,the main target of this paper is how to use ACL methods to improve the performance of motion control for ALVs.In this paper,a class of ACL control methods with the actor-critic framework is studied for solving lateral control problems,longitudinal control problems and cooperative adaptive cruise control problems of ALVs.The main research results and innovations in this paper are as follows:(1)Since the nonlinear and time-delay characteristics exist in the control system of ALVs,a synchronous iterative Dual Heuristic Programming(SI-DHP)algorithm is proposed for a class of affine nonlinear time-delay systems with control saturations.Due to the nonlinear characteristics of the established Hamilton-Jacobi-Bellman(HJB)equation,the SI-DHP algorithm is used to obtain the near-optimal solution and the convergence of SI-DHP is also analysed theoretically.The feedforward neural networks are utilized to design the actor and critic of SI-DHP,which are used to approximate the optimal value function and the optimal policy,respectively.Finally the performance between SI-DHP and the traditional PI state-feedback control method is compared via some typical control problems.The simulation results illustrate that SI-DHP has good capacity of self-learning and resisting disturbance.(2)A lateral control learning method based on SI-DHP is presented for solving the path tracking control problem of ALVs.The proposed lateral control method which has feedforward-feedback control structure uses the SI-DHP algorithm to optimize the steering control policy.The feedforward steering controller whose input signal is the curvature of the desired path outputs the feedforward steering control signal to track the desired curvature by considering the ackerman geometric steering model and the tire model.Both the time-delay and saturation characteristics of the steer control are considered during the designing procedure of the feedback steering controller.The SI-DHP algorithm is used to learn the feedback steering control policy by minimizing the lateral tracking error and energy cost.The lane change test,the circle road tracking test,the S-road tracking test and an urban road test are performed in the CarSim software to evaluate the proposed lateral control method.The simulation study illustrates that the proposed path tracking method based on SI-DHP has higher tracking precision compared with the traditional LQR control method and preview-based control method,especially at the limits of handling.Finally the performance of the proposed lateral control method is verified via the autonomous driving experiment of HQ3 on a highway road.(3)In order to obtain better learning efficiency,a parameterized batch actor-critic(PBAC)method is proposed in which the parameterized feature vectors based on kernels are learned from collected samples.In PBAC,the critic and actor which are used to approximate the value functions and policies share the same linear features.Since batched samples are used,the learning generalization ability is thus improved.Although the vehicle's longitudinal analytical model is unknown,the control policies which can adaptively tune the fuel/brake control signals to track different speeds are learned by PBAC to improve the control performance.The simulation and experiment studies are obtained using HQ7 to tracking different desired speeds from 5km/h to 40km/h.The simulation results are illustrated that the PBAC-based longitudinal control method has higher speed tracking precision compared with the traditional PI and learning-based PI control methods under a certain noise level.The efficiency of the proposed method is also verified by extensive experiments which are conducted on HQ7 while driving on flat,slippery,sloping and bumpy roads.(4)A cooperative adaptive cruise control(CACC)method based on online incremental dual heuristic programming(OI-DHP)algorithm is proposed.In the longitudinal cooperative control method,OI-DHP is used to learn the desired acceleration policy in order to track the target vehicle.In the lateral cooperative control method,the desired path is generated by using the posture relationship between the current vehicle and the target vehicle.Then the optimal tuning radius is obtained by evaluating the vehicle-path relationship according to the desired roads' boundary constrain.The simulation study illustrates that the longitudinal cooperative control method based on OI-DHP has a faster response and a smaller overshoot compared with the traditional LQR and PI control methods when the velocities of the target vehicle is step changed,suddenly changed or slowly changed.Besides,the tracking performance of the proposed CACC method is also evaluated in an urban road environment which is built in the PreScan software.The simulation results show that the proposed CACC method has good adaptive abilities under different road scenes.