Research Of Carbon-energy Combined-flow Reactive Power Optimization Based On Multi-agent Reinforcement Learning Algorithm

With the increasingly serious influence of the global warming, energy consumption industry pays more attention to the low carbon economy. As the largest carbon-emitting sectors, power industry will serve an important role in the development of low-carbon economy. In order to evaluate the carbon emissions accurately, we present a novel method about carbon emission flow in power systems. Considering the transmission characteristics of carbon emission flow and power flow in power grids, this paper proposes a reactive power optimization framework of carbon-energy combined-flow in which the power flow and carbon emission flow are treated as a whole, and the carbon emission responsibility is allocated among source, grid and load.As a member of the power systems, the power grid should pay for the carbon emission that belongs to it by computation. Different from the traditional reactive power optimization approaches, the carbon emission responsibility should be considered in the process of the carbon-energy combined-flow reactive power optimization, which means that both of the reduction of power loss and the decrease of carbon emission should be focused on during reactive power optimization in the power grid.When applying the reinforcement learning method to solve the problems, there is only one subject in which the iterative calculation is carried out, which results in a slow convergence rate. After the research of the reinforcement learning method and the swarm intelligence algorithms method, a kind on multi-agent reinforcement learning called PSO-Q(λ) is proposed. With the cooperative optimization of several agents, the convergence speed of multi-agent reinforcement learning is much faster than that of the single agent. In this paper, the algorithm is exploited to the reactive power optimization problem of carbon-energy combined-flow in small power grids. Simulation results show the effectiveness of the calculation, which lays the foundation for the collaborative optimization of the multi-agent reinforcement learning.With the expansion of the power systems, the control variable is increasing rapidly and the common algorithms of reinforcement learning suffer from the problem of ‘Curse of dimensionality’, which limits the development of other intelligent algorithms. To address this problem, this paper utilizes a method of Q matrix dimension reduction, which transforms a large-scale Q value matrix into a Q value matrix chain consisting of a number of small scale Q values matrix. While the method maintaining the internal relation of the variables, the state and the action space are greatly reduced, and the difficulty in searching is reduced. In addition, based on the idea of the imperialist competition algorithm, the imperialist competition Q-learning algorithm is proposed, which is a novel multi-agent reinforcement learning algorithm. Through the competition among multiple empires and the cooperative search and assimilation process in each empire, the optimal solution of the problem was obtained. The association memory characteristics of the cultural matrix makes the algorithm achieve fast optimization, which can be applied to resolve the reactive power optimization problem of carbon energy composite flow in large scale power systems. Finally, the simulation results show that the proposed algorithm still has an obvious speed advantage in the precondition of preserving the excellent convergence characteristics of the reinforcement learning, which provides a theoretical support for the online optimization of actual power grid.