Research On Behavior Control Of Autonomous Vehicle Based On Deep Reinforcement Learning

In recent years,with the continuous improvement of wireless communication,sensor positioning and automation and other technologies,the field of connected autonomous driving has been widely concerned by governments,enterprises,and universities around the world.Compared with human driving,connected autonomous driving has stronger perception ability and faster response-ability.Connected autonomous vehicles have great potential in improving urban traffic congestion,reducing traffic accidents,saving energy,and reducing emissions.But the traffic situation on real roads is complex and changeable.Therefore,the research of behavior control has always been the focus and difficulty of the research of networked automatic driving.Car-follow behavior and lane-change behavior are the two basic behaviors of vehicle driving,which have a very significant impact on road traffic safety and road traffic flow characteristics,and are also the research focus in the field of connected automatic driving.The traditional car-follow model either takes into account the information of only one car in front of it or does not fully consider the information of multiple cars before and after.Therefore,although the traditional Car-follow model can improve traffic efficiency and traffic stability,there are still certain limitations.The traditional lane-changing trajectory planning model lacks research on the coordinated trajectory planning of lane-changing vehicles and surrounding vehicles.And the existing lane-changing trajectory planning models based on deep reinforcement learning usually take steering,acceleration,or yaw angular acceleration parameters as the action space.The model needs to spend a lot of time learning to avoid planning obviously wrong trajectories.The training time of the model is long,the training of the model is difficult and it is easy to fall into the local optimum.In view of the above deficiencies,this paper establishes a behavior control model for connected autonomous vehicles,that is,a car-follow model considering the information of multiple vehicles before and after and a dynamic collaborative lane change trajectory planning model of vehicles that integrates reinforcement learning and trajectory curve equations.Firstly,this paper establishes a car-follow model based on deep reinforcement learning.Set the acceleration of the stalking vehicle to the action space of the model.Use the beforeand-after multi-vehicle information that mainly affects car-follow behavior as the state space of the model.The reward function is set to meet the goal of the road speed limit,collision avoidance,comfort,efficiency,and improvement of traffic flow stability.Then,this paper combines the deep reinforcement learning method and the transition curve equation to establish a collaborative dynamic lane change trajectory planning model.Set the acceleration of the lane-changing vehicle and the rear vehicle in the target lane and the cubic lane-change trajectory curve equation parameters of the lane-changing vehicle to the model action space.Vehicles that mainly affect lane change behavior are analyzed and their status information is set to the model state space.Set the reward function with the goal of maximizing the benefits of the two vehicles.Next,the simulation environment for training models and validation models was constructed using SUMO and Python,and the trajectory traceability verification simulation environment was constructed using the vehicle dynamics software Carsim.Finally,the training and verification results are analyzed.The simulation results show that compared with the traditional car-follow model,the carfollow model proposed in this paper considering the information of multiple vehicles in front of and behind has better safety in the case of emergency braking of the front car and effectively shortens the distance between the vehicles and improves the speed of the vehicle.In addition,the velocity disturbance decreases rapidly as it propagates in the fleet controlled by the model in this paper.This shows that the traffic stability of the model is good.The collaborative lanechange trajectory planning model proposed in this paper performs well in different lane change scenarios.When the surrounding traffic environment changes,lane-changing vehicles and the rear vehicles in the target lane respond in real-time.Efficiency and comfort are better than existing models.In addition,the Carsim simulation results show that the lane change trajectory planned by the model can be well realized by autonomous vehicles,and the vehicle driving stability is good during the lane change process.