Research On Bidding Strategy Of Generators In Electricity Market Based On Asynchronous Advantage Actor-Critic Reinforcement Learning

Promoting electricty trading system reform and improving market trading mechanism is one of the key tasks of the new power system reform.Since 2016,electricity trading centers have been set up one after another in China while long-term electricity market trading has been organized in Guangdong,Hunan,Yunnan and other provinces with expanding electricity.Meanwhile,artificial intelligence technology is at rapid growth,which will become a new engine for social and economic development.In May 2017,Alpha Go defeated World Go champion Li Shishi,marking the breakthrough of artificial intelligence technology.Reinforcement learning,core algorithm of Alpha Go,has received extensive attention and become a hotspot in theoretical research and application.Reinforcement learning has a strong ability of self-cognition and independent learning,which is able to tackle interactive and decision-making problems effectively,especially games in electricity market involving competition-cooperation relationship and market behavior of multi-agents.On the one hand,electricity market bidding based on reinforcement learning guarantees reasonable interests of generators in market trading and realization of intelligent bidding decision-making.On the other hand,it promotes effective market supervision with electricity trading centers,which resulting in market efficiency improvement and resource allocation optimization.On the background of continuous construction of electricity market and rapid development of artificial intelligence,reinforcement learning is applied to the research of generators' bidding strategy in long-term electricity market in this paper.First of all,current situation of electricity market bidding and reinforcement learning is analyzed,the characteristics of traditional bidding methods and reinforcement learning are summarized and main research framework of this paper is put forward.Secondly,structure system of long-term electricity market is proposed on the aspect of primary market,secondary market and unbalanced market,and trading organization of long-term electricity market and process of monthly centralized bidding trading are analyzed.Next,basic framework of reinforcement learning,Markov decision-making process,is proposed,principles and algorithms of reinforcement learning based on value function,strategy gradient and Actor-Critic are summarized,and characteristics and applicability of different methods are analyzed.Then,strategy network model Actor and value network model Critic of generators,clearing model of centralized bidding trading and bidding model of generators based on algorithm of asynchronous advantage actor-critic are constructed,and overall structure of generator bidding model in long-term electricity market is put forward.Finally,basic parameters of long-term electricity market simulation are set up,and simulations of long-term electricity market with single-agent and multi-agent are carried out.According to the simulation results,bidding strategy of generators and market operation status are analyzed.The results show that in ong-term electricity market,generators prefer physical withholding than economic withholding.The larger market power generator has,the higher the probability of physical withholding is.The closer the price deviation is to 0,the larger the proportion of physical withholding is.Market clearing price rise and market efficiency is reduced when low-cost generator withholds large amounts of electricity.Therefore,electricity trading centers should focus on the supervision and punishment of physical withholding of generators with large market power and low generation costs.