Using Reinforcement Learning To Study The Features Of The Participants’ Behavior In Wholesale Power Market

In the traditional electrical market models, the power systems were implemented by thevertical integration of management, and the power companies monopolized all aspects ofelectricity production. These models can not be conducive to the healthy and sustainabledevelopment of the power industry in the long time. In the1990s, Chile was the first countryto break the power monopoly and introduced the market mechanism. Some countries inEurope and Latin America established the power market after that very quickly. Some of themare the PJM, CaISO and NYISO electricity market in United States of America, the NETAelectricity market in United Kingdom and the Nordic electricity exchange Nord Pool Spotfound in Denmark, Finland, Sweden, Norway, Estonia and Lithuania, etc. Nowadays, a powermarket has become the major trend in the electrical system reform in some countriesnowadays. The existing research results and practical applications have proven that throughthe interaction between the participants in the competitive electricity market, the systemefficiency increases significantly. This reduces the production costs and improves the qualityof electricity service. Based on the abundant experiences from the market economy in somedeveloped countries, the electricity market has also given many effective and basis theories.The electricity market in South America is a proven fact that the developing countries can alsodevelop for themselves the appropriate electricity market. In recent years, Asian countrieshave also carried out the electricity market researches, including my country–Vietnam,which also actively promotes the electricity market.In the market environment, the participants constantly optimize their bidding strategies inorder to obtain the highest profits. Not only each participant estimates the opponent’sstrategies but also evaluates its market position, which is the basis for his/her setting out theappropriate strategies. Therefore, the participants do not have the entire market information,and they have more difficulty to optimize their bidding strategies. As the market scale hasincessant expansions, the market environment at present has become more complex. Thus,knowing how to find out the optimal bidding strategy for the participants through an efficientlearning method is very essential and important. In this dissertation, we study thereinforcement learning method and apply it in the wholesale power market. The detailedcontents of the research are presented from chapter2to chapter6of the dissertation. Thetenors of the dissertation are summarized as follows:(1) The mathematical models for the wholesale power market are recommended,including the structure of the day-ahead wholesale power market model, the dynamic power market model, the running processes and the participants’ cost function, etc.(2) The basic theories of the reinforcement learning algorithm are researched includingthe origin form of the reinforcement learning, the RE-reinforcement learning method, the Q-learning algorithm, the Fuzzy Q-learning algorithm and the SA-Q learning algorithm.(3) The Q-learning algorithm is applied for the power suppliers in the day-aheadwholesale power market to search for the optimal bidding strategy. The feasibility of themethod is sustained through multiple simulations.(4) The Fuzzy-Q learning algorithm is applied for the power suppliers in the day-aheadwholesale power market. The algorithm combined the fuzzy logic theory with the Q-learningalgorithm to enhance the adaptability of the algorithm. The feasibility of the method isvalidated through multiple simulations.(5) The SA-Q learning algorithm is applied in the dynamic power market, where allparticipants, sellers and buyers, could change the bidding strategies per hours. The algorithmcombined the solid annealing algorithm with the Q-learning algorithm to enhance theconvergence of the algorithm. The validation of the method is demonstrated throughoutmultiple simulations.This dissertation focuses on the reinforcement learning algorithm to expand the study ofthe wholesale power market. The participants have enough strong instruments to optimizetheir own bidding strategies. Currently in China and Vietnam, the application of thereinforcement learning for the electricity market has been repeatedly raised by manyresearchers. However, it still lacks the depth research. The main result of the dissertation is acomparison and optimization of various methods of reinforcement learning algorithm. Someinnovative aspects could be listed as below:(1) Analyze the feature of the participants’ behavior in the wholesale power market. Aday-ahead wholesale power market using DC-OPF solution to determine the active powerunits at each node in the power system is proposed. The participants (including GenCos andLSEs) with offer function and bid function are analyzed. The reinforcement learningalgorithm is proposed to apply for the participants to optimize their own bidding strategies.(2) Establishing the mathematical model and the basis theory for the wholesale powermarket based on the reinforcement-learning algorithm. While implementing the Q-learning forthe suppliers in day-ahead wholesale power market model, the state variables such as the state,the action and the reward variable are required. So, the equations of the state and rewardvariable and the action selection method are suggested. The ε-greedy method and theBoltzmann exploration method is also integrated in the Q-learning algorithm.(3) The optimal parameters are different during process simulation for each different algorithm. In the application of the Q-learning algorithm, the coefficients are the randomconstants. In the application of the Fuzzy Q-learning algorithm, the coefficients are the outputsignals of the fuzzy block, and the input signals are the market information. To identify themarket power, two indices are proposed to evaluate the power supplier’s market. These arethe price index and potential market index.(4) Analyzing and comparing the strong points and/or weaknesses of each reinforcement-learning algorithm.(5) Based on the actual demand, we would like to propose the dynamic power marketmodel. In this model, the GenCos and LSEs could change their bidding for day-ahead at eachhour. The bidding function is a block ladder form. This gives participants a simplification inthe bidding work.Finally, in this dissertation, we propose the day-ahead wholesale power market model, inwhich electricity prices are cleared through the uniform pricing rule, and the ISO determinesthe active power units at each node on the power system by the DC-optimal power flow. Thetarget of the participants is maximizing the profit, and to achieve the target, they apply manydifferent algorithms including the reinforcement learning algorithms. In the dissertation, wechoice the reinforcement learning algorithms as the main algorithm to analyze and apply forthe participants. The model was modified based on the day-ahead wholesale power marketunder the double-sided auction market. Besides, in the thesis, varieties of the reinforcementlearning algorithms, including Q-learning, fuzzy Q-learning, and SA-Q learning algorithm arepresented. The simulation results demonstrate that the Q-learning algorithm is alwaysconvergent. The fuzzy Q-learning algorithm converges more rapidly than the conventional Q-learning and the suppliers can choose the best bidding strategy according to the changingpower market structure. The simulation results also demonstrate that the SA-Q learningalgorithm has the best convergence. To prove the proposed theories and the recommendedmodels, the simulations of the IEEE6-busbar, IEEE9-busbar and IEEE30-busbar for thepower transmission network were carried out. All simulations were simulated based on theMAPPOWER package tools under the Matlab environment. The encouraging results in thethesis are fundamental for us that could be more involved in the electricity reforms and smartgrid researches.