Load Modeling Of Distribution Network Based On Hierarchical And Cooperative Reinforcement Learning

The distribution network plays an important role in the power system with the function of distributing electricity.A large number of distributed renewable energy sources and various types of loads are connected to the distribution network,which makes the load characteristics of the distribution network more complicated and makes the load modeling of the distribution network more intractable.How to quickly and accurately model the load in the distribution network under the time-varying and random environment has become an urgent problem in the power system field.The distributed renewable energy and loads in the distribution network have strong randomness,time-variation and complexity.Firstly,this thesis models a specific load component in the distribution network under uncertain environments.Then,this thesis studies a type of load with static characteristics of time-varying voltage.Finally,the dynamic equivalent modeling of the composite load in the active distribution network is carried out.The research objects range from static to dynamic,from single load model to multiple load model.In order to deal with the problem of load modeling in various uncertain environments,this thesis proposes a variety of hierarchical and cooperative reinforcement learning algorithms based on classic Q-learning to meet the real-time and accuracy requirements of load modeling in the distribution network.These algorithms range from simple to complex,from discrete variables to continuous variables,from addressing low-dimensional simple problems to high-dimensional complex problems.The proposed algorithms correspond one-to-one with the research object and are closely integrated.Specifically,this thesis conducts the following three researches:Firstly,aiming at the specific load component of the rapidly developing electric vehicle in the new distribution network,a multi-step Q(λ)algorithm based on multi-agent cooperation(MACQ(λ))is proposed to model the electric vehicle charging load.This thesis utilizes multiagent technology to build different types of agents,and studies the interaction between the agent and the environment and the interaction among agents.In order to accurately describe the uncertainty of electric vehicle behaviors and charging load characteristics in the distribution network,this thesis adopts the MACQ(λ)algorithm by introducing eligibility traces and communication learning mechanisms.The proposed algorithm can provide optimal behavioral decisions for electric vehicles,and obtain the charging loads with the spatial-temporal distribution.The case studies are carried out on the actual planning map and the 10 k V distribution network of a city in China.The simulation results verify the feasibility of the modeling for electric vehicle charging load based on the MACQ(λ)algorithm.Besides,the influence of different power grid pricing mechanisms and shift strategies on the electric vehicle charging loads with spatial-temporal distribution and on the voltage and network loss of the distribution network are studied.Secondly,aiming at the load model with static characteristics of time-varying voltage,a hierarchical area-load modeling framework is proposed,which can accurately reflect the interaction between the internal and external power networks.Furthermore,an adaptive reinforcement learning algorithm based on multi-agent cooperation(MACSARL)is proposed to perform online parameter identification of the equivalent model.The upper-level problem of this hierarchical framework is an area-load equivalent modeling.In the equivalence model,a novel load model that considers the interaction error of power flow is constructed according to the model predictive control theory.In addition,this thesis proposes a weighting strategy with time attenuation to distinguish the contribution of sampling points at different times in the sliding time window to the equivalent model,so that the obtained equivalent model can better reflect the current load characteristics and obtain more accurate optimal power flow.The lower-level problem of the hierarchical framework is a security-constrained optimal power flow problem based on the time-varying equivalent load model of the external network,so as to consider the influence of the internal and external networks.Mathematically,the proposed area-load equivalent model is a complex nonlinear optimization problem with multiple local optimal solutions,and its decision variables are multi-dimensional continuous types.In order to accurately solve the problem,this thesis proposes a novel adaptive reinforcement learning algorithm based on adaptive boundary search and variable learning factor strategies.The algorithm utilizes a multiagent cooperation mechanism to identify the area-load equivalent model online.Simulation results show that the proposed equivalent model has higher accuracy than the existing model.The MACSARL algorithm has a significant improvement in computational accuracy and ability to track time-varying parameters compared with least squares and FORL algorithms.Finally,aiming at the composite load in the distribution network integrated with the renewable energy sources,a hierarchical deep Q network(HDQN)algorithm is proposed for the dynamic equivalent modeling of the distribution network.The distribution network contains a variety of sources and loads with different characteristics,such as ZIP loads with static voltage characteristics,induction motors with dynamic characteristics,and constant-speed and constant-frequency wind turbines.In order to solve the problem of time-varying and accuracy of the equivalence model,multiple load models are used for dynamic equivalent modeling of the active distribution network.Then,the HDQN algorithm is utilized to select the equivalent model scheme and determine the model weights,so as to obtain the equivalent power output of the equivalent load model of the distribution network.The state of reinforcement learning is a multi-dimensional discrete-time sequence composed of continuous variables.The proposed algorithm adopts the long short-term memory(LSTM)neural network to extract features from the input time sequence signal.In terms of solving algorithms,the strategies such as prioritized experience replay and Huber loss function are utilized to improve the DQN algorithm.Simulation results illustrate that the proposed HDQN algorithm can effectively perform the dynamic equivalent load modeling of the active distribution network integrated with the renewable energy sources.The accuracy of the proposed algorithm in the equivalent active load is 3 times that of the traditional DQN algorithm.