Cooperative Distributed Q-Learning For Traffic Signal Timing Optimization With Clustered Data Collection

:With the acceleration of urbanization in the world, urban traffic congestion has become the urgent challenge in cities. As the basis for information carrying platform of intelligent transportation systems, Vehicle Ad-hoc Networks (VANET) has become one of the focus researches of intelligent transportation technology. It is fruitful for providing convenient transportation guide service by using real-time traffic information based on VANET to optimize the intersections’signal timing. Aimed at the characteristics of highly dynamic with topology of VANET and continuous non-stationary of the traffic flow, two key problems of VANET clustering traffic data collection and traffic signal timing optimization are studied. The objective of this paper is to reduce urban congestion, improve road utilization.Firstly, to enhance the stability of network topology, improve packet delivery rate, and reduce communication delay and overhead, a dynamic clustering traffic data collection algorithm is proposed. According to the highly dynamic characteristics of the VANET vehicle nodes, the affinity propagation algorithm is used to elected cluster head. A stable cluster structure which is suitable for VANET road traffic scenarios is produced by calculating number of neighboring nodes, speed, distance, lane weight of the cluster members. At last, to reduce the instability of the cluster structure and increase clusters’life time, a cluster maintenance method is introduced. The traffic status information which obtained from clustering algorithm is transmitted to the intersection agents based on V2I communication. It is provide efficient and accurate traffic state information for traffic signal timing optimization.Secondly, in view of non-continuous, time-varying and random traffic flow in large scale urban environment, a distributed cooperative Q-learning with fast gradient descent function approximation algorithm for signal timing optimization is proposed. A clustering data collection based intersection model is designed to estimate vehicle queue length at intersections of each lane. And then the Q-learning model is built to describe traffic signal timing optimization. By employing cooperative behaviors with neighboring intersections, the optimal policy without any central supervising agents is achieved. To address the curse of dimensionality effectively, the Q-learning function is approximated by using a fast gradient-descent method. To seek exploration-exploitation balanced strategy and accelerate the convergence rate, an improved ε-greedy strategy in Q-learning is applied.Numerical simulations driven by VanetMobiSim, NS-2, GLD and MATLAB demonstrate that the proposed approach can be effective.