Adaptive Signal Control Based On Deep Reinforcement Learning

Urban traffic exists in all areas of urban public space.After long-term development,it has formed a more perfect pattern and improved people's travel efficiency.However,with the development of economy and the acceleration of urbanization,the urban population and the number of the automobiles have increased rapidly.As a result,traffic congestion is becoming more and more serious,traffic congestion,transport inefficiency,environmental pollution and other issues are becoming increasingly serious,which seriously affects the sustainable development of the city.The adaptive control of traffic lights can effectively alleviate traffic congestion.However,the traditional traffic lights control methods usually optimize the timing schemes of traffic lights according to the traditional traffic parameters such as queue length,traffic flow,lane occupancy ratio and so on.They do not make full use of intersection status information,or only consider the optimization of single intersection signal lights,and do not cooperate with other intersection signal lights to achieve the optimal control of regional road network.In order to solve the above problems,the adaptive signal control based on deep reinforcement learning is studied in this paper.The main work is as follows:(1)This paper proposes a cooperative deep Q-network with Q-value transfer(QT-CDQN)for adaptive multi-intersection signal control.In QT-CDQN,a multi-intersection traffic network in a region is modeled as a multi-agent reinforcement learning system.Each agent searches the optimal strategy to control an intersection by a deep Q-network that takes the discrete state encoding of traffic information as the network inputs.To work cooperatively,the agent considers the influence of the latest actions of its adjacencies in the process of policy learning.Specially,the optimal Q-values of the neighbor agents at the latest time step are transferred to the loss function of the Q-network.The advantages of QT-CDQN lie not only in the effectiveness and scalability for multi-intersection system,but also in the versatility to deal with heterogeneous intersection structures.The effectiveness,adaptability and scalability of the proposed algorithm are verified by experiments from various angles.(2)This paper proposes a multi-task deep Q-network with Q-value transfer(QT-MTDQN)for adaptive multi-intersection signal control.Traffic flow of intersections in regional road network is usually different,so the control of intersections in road network can be regarded as different tasks.The control of each intersection corresponds to one task.There are similarities among multiple tasks.By sharing the representation of similar tasks,the model can have better feature extraction ability,decision-making ability and generalization ability.Firstly,an expert DQN network is trained for each intersection.Then a multi-task DQN network is trained under the guidance of multiple expert networks,which enables the multi-task network to learn how to work in multiple tasks at the same time.Then the trained multi-task network can extend the knowledge to new tasks without expert guidance(intersection of different traffic flow densities).Finally,the multi-task network is transferred to each intersection by using transfer learning technology,and then the cooperative algorithm based on Q-value transfer is used to control the signal lights of multi-intersection cooperatively.The experimental results show the effectiveness of this method.The comparison between multi-task learning and non-multi-task learning proves that multi-task learning does improve the performance of this method.