Autonomous Driving Decision Making And Planning For Intelligent Vehicles Using Reinforcement Learning

Reinforcement learning is one of the most important areas in machine learning.Previous tabular algorithms are difficult to solve reinforcement learning problems with large scale or continuous state space.Therefore,the value function approximation(VFA)methods for reinforcement learning have been widely studied in recent years.One of the key problems for VFA is feature representation,which has a direct influence on the algorithm performance.Supported by the National Natural Science Foundation of China(NSFC),this paper is focused on the reinforcement learning and its applications in driving decision making and path planning for intelligent vehicles,emphasizing on the feature representation in large-scale state space,the driving decision making in dynamic traffic flow as well as the hierarchical path planning based on reinforcement learning.In this paper,the development of reinforcement learning is reviewed,the research propress of intelligent vehicles is introduced,and the driving decision making and path planning methods for intelligent vehicles are summarized.The main contributions of this paper are as following:(1)A random-neuron based approximate policy iteration(RN-API)algorithm is proposed for reinforcement learning with large scale or continuous state space.Inspired by the extreme learning machine(ELM),the RN-API algorithm approximates the value function by using a feedforward neural network with random hidden neurons,in which the input weights and hidden layer biases are assigned at random and only the output weights need to be learned.The RN-API algorithm not only guarantees the generalization performance,but also improves the feasibility of the algorithm by reducing the number of tunable parameters for performance optimization.Comprehensive simulation studies on two benchmark learning control problems are carried out to test and compare the performances of approximate policy iteration(API)algorithms with different feature construction methods.The results show that the RN-API algorithm can obtain comparable or better performance with fewer tunable parameters being optimized manually in the feature representation process compared to other API algorithms.(2)A driving decision making method using API is proposed for intelligent vehicles in dynamic traffic flow.In the proposed method,the driving decision making problem is modeled as a Markov decision process(MDP).Then,the API is used to learn the optimal or near-optimal driving decision making policy.A driving decision making simulation platform based on a 14-DOF model of the autonomous vehicle HQ3 is introduced,on which the performance of the proposed method is evaluated.The simulation results demonstrate the feasibility and validity of the proposed method.(3)A hierarchical path planning approach based on reinforcement learning for intelligent vehicles is proposed with a two-level structure.In the first level,the A* algorithm is used to quickly acquire a geometric path and several points are selected as sub-goals for the next level.In the second level,an approximate policy iteration algorithm is used to learn a near-optimal local planning policy with kinematic constraints.Using this near-optimal local planning policy,the intelligent vehicle can find an optimized path by sequentially approaching the sub-goals obtained in the first level.The training of this local path optimizer can efficiently use sample experiences collected randomly from any reasonable sampling distribution.Furthermore,the reinforcement learning based local path optimizer has the ability of dealing with the uncertainties in the environment.Simulations for path planning in various types of environment have been carried out and the results demonstrate the validity and efficiency of the proposed approach.(4)A driving decision making system based on reinforcement learning for intelligent vehicles is designed and implemented.The performance of the driving decision making system is tested in real highway traffic environment.The results show that the driving decision making system based on reinforcement learning can ensure a safe running for the intelligent vehicle HQ3 using the decision policy learned form samples,which laid a foundation for further improving the autonomous ability of the intelligent vehicle in the future.At the end of this paper,the problems to be further studied in the future are analyzed and summarized.