Application Of Deep Reinforcement Learning For The Train-to-Train Communication Based Train Control System

It is imperative for urban rail transit in China to develop towards intelligentization.The shortcomings such as a huge amount of wayside devices,the complex interfaces and the large train-ground communication delay have been exposed in the long-term operation of the communication based train control(CBTC)system.These existing shortcomings affect system performance and increase system maintenance costs.The application of train to train(T2T)communication in urban rail transit is expected to solve the shortcomings of CBTC system.Therefore,this paper focuses on the train control system based on T2 T communication,and adopts deep reinforcement learning to make the train more "intelligent".The train can complete the resource allocation and train control by itself,and the whole system has better performance and more flexible operation mode,thus contributing to the intelligent development of urban rail transit.The main work and innovations of this paper are as follows:(1)A new train control system based on T2 T communication is proposed,which adopts LTE-M(Long Term Evolution for Metro)system.However,train to wayside(T2W)communication coexists with T2 T communication in this new train control system,which results in the strain on frequency resources.To make full use of limited spectrum resources,frequency reuse is adopted as an efficient technique,however,it brings the co-channel interference unfortunately,which affects the quality of service(Qo S)for T2 T and T2 W users.Thus,this paper proposes a multi-agent deep reinforcement learning(MADRL)based autonomous channel selection and transmission power selection algorithm for T2 T communication to reduce the co-channel interference.Specifically,the transmitter of each train is regarded as an agent,which interacts with the environment and selects actions to implement a distributed resource allocation mechanism independently.Simulation results show the superiority of the proposed algorithm: the system throughput and the successful transmission probability of T2 T links are greatly improved compared with the random allocation scheme and the existing resource allocation schemes for T2 T communication.(2)The train cooperative formation technology is proposed to realize a more flexible train operation mode in the proposed new train control system.For the problem of train control in the train cooperative formation,this paper adopts deep Q-learning algorithm based on pre-exploration to realize autonomous control of the train,which breaks the limitation of centralized control of the train in the traditional CBTC system.The pre-exploration not only accelerates the convergence speed of the proposed algorithm,but also improves the training effect.In addition,considering the passengers experience,the algorithm is improved and two optimization measures are taken.First,when the train reaches the control target,the acceleration output by the algorithm is set to 0.Furthermore,the state and reward function in the algorithm are improved.The simulation results show that the optimized algorithm has a smoother acceleration curve during the train control process,so it can improve the comforts of the passengers.Moreover,the proposed algorithm can accomplish the train control task well in the two or three trains tracking scenario and meet the control requirements of the train cooperative formation system.Finally,this paper points out the deficiencies in the research content and how to improve them.