Deep Deterministic Policy Gradient-based Smart Generation Control And Coordination Technology

With the development of integrated energy systems(IES),the interaction of various heterogeneous energy subsystems as well as the constant infiltration of large-scale renewable energy,the random disturbance power in the interconnected grid will become more frequent,thereby intensifying the pressure for the regulation and balance of the frequency.Further,the performance-based frequency regulation market and the competition between power grid operators which it entails demand greater coordination between different modules,different objectives,and more complex units in automatic generation control(AGC)systems.The traditional AGC strategy cannot satisfy the power grid requirements for highly controlled frequency regulation control.A critical scientific problem facing electricity grid engineers is the need to enhance the self-learning and collaborative learning capabilities of AGC strategies in order to realize satisfactory frequency regulation control performance of power grids.In this paper,five coordination problems in grids are addressed by identifying ways of improving and utilizing the collaborative learning capability of the deep deterministic policy gradient algorithm,and by employing the following data-driven coordination strategies:1)Multi-objective coordinated optimization of single-area AGC power distributor: In order to solve the multi-objective coordination problem which undermines the control performance and economy of AGC power generation power command,an integrated energy system automatic generation control(IES-AGC)that models the demands on electricity,natural gas and heat is firstly established.Furthermore,a more robust multi experience probabilistic replay twin-delay deep deterministic policy gradient(MEPR-TD3)algorithm is proposed to solve the problem of AGC generation power command dispatch,which not only can reduce the area control error and the regulation mileage payment,but also satisfy the multi-energy flow balance constraint.2)Coordination control between single-area AGC controller and power distributor: aiming at the coordination problem of frequency regulation resource waste,unit repetitive regulation and frequency collapse,a framework called the integrated generation control power dispatch(IGCPD)is firstly designed,which integrates independent controller and power distributor into an agent,thereby fundamentally solving the non-coordination problem caused by two independent modules.Moreover,by reasonably designing reward function,the framework improves the dynamic performance,economy and regulation capacity of the fast unit,thus preventing frequency collapse.In addition,a swarm intelligence based large-scale deep deterministic policy gradient(SI-DDPG)algorithm is proposed for the framework,which uses multiple agents with different reinforcement learning principles to carry out distributed optimization,so that a more robust IGCPD strategy can be obtained.3)Multi-objective coordination control between multi-area AGC controllers: a distributed intelligent coordination automatic generation control(DIC-AGC)framework which addresses the coordination control problem of multi-area AGC controllers in the performance-based frequency regulation market is proposed.The framework considers the effects of performancebased frequency regulation market mechanism on tie power control and frequency control,and also balances the requirements of different power grid operators.In addition,an evolution imitation curriculum-based multi-agent deep deterministic policy gradient(EIC-MA4DPG)algorithm is also put forward.The algorithm design is rooted in the concepts of imitation learning and curriculum learning,thus delivering a coordination control strategy of greater robustness and adaptive decision-making ability.Given reasonable design parameters for the training of the algorithm,which in turn is guided by demonstration samples,each controller can realize optimal coordination control simply by observing the state of its own area.4)Distributed coordination problems between multi-area AGC controllers and power distributor: These coordination problems can lead to tie power fluctuation and increased regulation payment.In consideration of the complex mechanisms and multitude of agents involved,this paper proposes an elaborate strategy,titled ‘octopus’ coordinated automatic generation control(OC-AGC),which is characterized by a cloud collaborative framework mimicking the nervous system in the octopus,wherein the controller and power distributor in each area are treated as independent agents which output the power generation commands.Further,a track-explore distributed multi-agent deep deterministic policy gradient algorithm(TED-MADDPG)is put forward for this framework,whereby various techniques such as trackexplore and integrated learning strategy guide the convergence of the algorithm and prevent the algorithm from falling into the local optimum,thus realizing a more robust global optimal decision.Finally,the agent is capable of making a global optimization decision based on its own area state.5)Multi-objective coordination of emergency frequency control and AGC strategy: In order to address interconnected grid derivative accidents and coordination problems between the emergency frequency control and AGC,a wide-area AGC(WA-AGC)framework which can be coordinated with the emergency control strategy is presented.The real-time measuring data in the wide-area monitoring system(WAMS)is adopted in order to determine grid power flow distribution while carrying out frequency regulation.Then,the traditional AGC power dispatch strategy is divided into four control intervals according to the current state of the system,while considering the dynamic performance,economy,system security and economy of the frequency regulation,so as to match the emergency frequency control strategy.Then,in order to solve these multi-agents,multi-time-scale,and multi-objective complex coordination problems,a supreme agent exploration distributed multiple delayed deep policy gradient algorithm(SAE-MD3)is raised for the framework.The algorithm takes advantage of different exploration strategies for distributed exploration so as to obtain a more robust AGC generation power command dispatch strategy.In addition,this paper proposes a wide-area frequency control strategy(WA-FCS)from the perspective of frequency stability control.The wide-area frequency control strategy organically combines wide-area AGC and emergency frequency control,and reduces the amount of loads/units cutting in emergency frequency control via a centralized hierarchical training strategy in order to prevent the occurrence of derivative accidents such as power line overload.Since wide-area frequency control addresses a complex grid environment with large random disturbances and emergency faults,in order to enhance the robustness and multi-task learning capability of the wide-area frequency control strategy,this paper introduces a distributed metadeep deterministic policy gradient(DM-DDPG)algorithm to obtain a wide-area frequency control strategy with more collaborative learning capability.6)Finally,this paper demonstrates a parallel system-based AGC simulation platform by applying it in a county grid demonstration project.