Research Of Vehicle Cooperative Adaptive Cruise Control System Based On Reinforcement Learning

In this paper proposed a new method of Cooperative Adaptive Cruise Control(CACC)and designed a model to solve the CACC problem from the perspective of computer science.Different from the traditional PID control method,this paper proposed the reinforcement learning method to solve this problem.Each vehicle in the system had been designed as an independent agent,and any number of vehicle agents within the communication range can be add to the vehicle platoon.The experimental part of this paper used five vehicles model.This thesis goes from the elementary to the profound.First,the Q-Learning method is used in building a model for researching in CACC problem.For the restriction of Q-value which is stored in the table for the algorithm,we discrete the states and actions,the discrete precision is one.When designing the reward,the relationship between the distance which between two cars and the expected safe headway in the current and next state is being considered,this is the basic design method of the reward algorithm.The headway strategy mentioned is used to calculate the relative speed between the front car and rear car to expect safe headway.During the learning session,?-greedy strategy is used in the selection of action,and the relationship based on the real distance between two cars and the expected safe headway is used to guide the selection of action.It comes up with rules about how to select action in the heuristic ?-greedy algorithm,and it will prune the selection to speed up the learning process in order to avoid choosing the useless action.In the Q-learning method training process of CACC,the variance of the reward curve was found to be particularly large.It is considered that the discrete precision is not enough that cause errors in state and action.After the error is accumulated,the curve fluctuates greatly.Therefore,the Deep Q-Learning is introduced into CACC.The above analysis of QLearning showed that the Q-value representation in the Q-Learning method is implemented through table storage.The Q-value table cannot be too large under the tradeoffs of operating efficiency and table storage.In view of the nature of the universal approximation of neural networks,the neural network is used to approximate the Q-value function of the learning algorithm.The neural network is introduced into Q-Learning algorithm through the value function approximation method to form Deep Q-Learning algorithm.The Deep Q-Learningalgorithm avoids the Q-value table of the Q-Learning algorithm and can accommodate more state and action.In this case,the discrete precision of the states and actions was set to 0.1,and the state dimension is also added,and the front vehicle acceleration information is incorporated into the state variables.The design idea of the reward function in the CACC platoon control algorithm based on Deep Q-Learning is consistent with that in the CACC platoon control algorithm using QLearning,but the strategy in which the safety car distance is expected to use the distance strategy takes into account the judgment of the acceleration of the vehicle in front of the sports trend.After training,the curve of reward and learning rounds is more reasonable.In comparison tests,both CACC methods exhibited synergistic advantages.Vehicles in the fleet system responded more consistently to changes in movement trends.However,the Q-Learning-based method is “harder” than the deep Q-Learning-based method.Therefore,it can also indicate that the Deep Q-Learning-based CACC platoon control method is closer to the actual application.However,the CACC platoon control method based on Deep Q-Learning in this paper has its disadvantages.After all,vehicle control is a continuous control problem.Discrepancies between states and actions will generate errors.Therefore,the following research will be based on continuous Controlled reinforcement learning methods.