Research On Automatic Train Operation Based On Reinforcement Learning

In recent years,rail-based transit systems have been playing an increasingly key role in national comprehensive transportation system.However,with the shortening of operative headway and the shifting of train operation speed,manual driving train is unable to meet the practical needs.Meanwhile,the development of Automatic Train Operation system(ATO)is becoming the general trend all over the world.Most of the traditional automatic train operation methods focus on the driving strategies with the given model or fixed parameters,which are inefficient in robustness,adaptability and self-learning abilities when facing complex operation environment and diverse operation needs.While new Artificial Intelligence(AI)technologies,which could make ATO more intelligent and wiser,are difficult to be applied in traditional methods due to the given model or fixed parameters.On the other hand,the basic theory and technology application of AI have developed rapidly in decades.Among the AI technologies,reinforcement learning and deep reinforcement learning have the outstanding performances not only in image recognition,semantic recognition,game competition and other aspects of hotspot fields,but also in the traditional automatic control field,such as bionic robots,driverless vehicles and so on,which have brought new opportunities in the correlational fields.According to the insightful cognition and analysis of railway automatic train operation technology,this thesis explores the issue of automatic train operation methods based on reinforcement learning,in which the AI methods such as reinforcement learning are studied to achieve the key requirements of railway transportation,which are safety,punctuality,energy-efficiency and comfort.To be specific,the main innovations and contributions of this thesis are listed as follow.Firstly,on the basis of the control principle of ATO and the driving experiences of excellent drivers,a policy-based reinforcement learning algorithm for intelligent train control is proposed to reduce the energy consumption of train traction.Through the establishment of train control expert system,the train control strategy which meets the requirements of energy-saving conditions and passenger comfort is constructed.Based on the expert system,neural network is used as the train driving controller,and a policybased reinforcement learning algorithm is presented to optimize the parameters of neural network and the control strategy through continuous learning of the historical data of train operation,so as to achieve better flexibility to the changing operation scenarios.A case study conducted based on the real-world data illustrates the effectiveness of the proposed approach.The simulation results indicate that compared with the current ATO control algorithm,the proposed algorithm has a better energy-efficient and punctual performance.Secondly,an optimization method of automatic train operation based on Q-learning,which is an off-policy algorithm of reinforcement learning,is proposed.Aiming at the traction energy consumption and punctuality in the whole process of high-speed railway station operation,the constraints of line speed limit,ramp,and braking force and traction limit are considered to establish the energy-saving driving control model of railway.Then the model proposed is transformed into multi-step decision-making optimization problem.According to the two goals of train punctuality and energy-saving,the corresponding value function table is established,and a Q-learning-based railway energy-saving driving strategy optimization algorithm is designed.In order to verify the effectiveness of the proposed method,the energy-efficiency optimization strategy proposed in this section is simulated and verified by taking the actual train and line data.The results show that the algorithm can effectively reduce the energy consumption between railway stations under all constraints.Finally,with the introduction of deep reinforcement learning(DRL)method,a train operation control optimization strategy based on the deep Q-network(DQN)method is proposed.In this method,instead of the traditional train dynamics prior knowledge and the pre-designed velocity profile,a large number of driving experiences are applied to train deep reinforcement learning neural network.The optimization model takes the energy consumption as the main objective,with the constraint on speed restrictions,punctual arrival,passenger comfort and precise stopping,which achieves an "end-to-end" intelligent vehicle control approach.Based on the real-world data,some cases are conducted to illustrate the effectiveness,adaptability and robustness for the changing train schedules of the proposed approach.