Research On Distributed Energy Management In Smart Grid Based On Multi-Agent Reinforcement Learning

With continuous advancement of the implementation of the “Carbon Peak and Neutrality”goal,technologies such as renewable energy,distributed power generation,and demand response have developed rapidly.The smart grid continuously integrates advanced technology applications through power flow,information flow,and business flow,and establishes a widely distributed energy transmission network.With the rapid expansion of power grid,regional power grid is no longer controlled by a single entity,and traditional centralized power system is also facing challenges.Distributed control methods have received extensive attention due to their high scalability,good privacy,and low maintenance costs.In order to realize the efficient and reliable operation of the smart grid,distributed energy management has become an important issue in the distributed control of new power systems.However,in the face of the optimization problem modeling that is more suitable for the actual situation,the existing distributed energy management algorithms are neither able to address optimization objectives with coupled or non convex terms,nor can they effectively cope with the uncertainty brought about by renewable energy generation and real-time demand,which has become a new moment to introduce artificial intelligence technology into smart grid distributed energy management.As a new distributed decision-making method,multi-agent reinforcement learning is superior in dealing with nonlinear goals and state uncertainties,but there are still two issues to address in applying multi-agent reinforcement learning to smart grid distributed energy management: First,the existing multi-agent reinforcement learning algorithms all adopt the framework of centralized training and distributed execution,which is difficult to deploy for fully distributed power network nodes; secondly,it is difficult for multi-agent reinforcement learning algorithms to deal with the energy balance constraints of the distributed energy management problem.Simply converting the optimization problem constraints into the penalty term of the learning objective is likely to bring sparse rewards.Based on the above challenges,this paper proposes fully distributed multi-agent reinforcement learning algorithms under different reinforcement learning frameworks.Aiming at distributed energy management problems considering non-convex goals and uncertainty,this paper provides corresponding algorithms and performance analysis.The specific work content includes the following aspects:1.Research on fully distributed multi-agent reinforcement learning algorithm based on Q-learning.Aiming at the convergence requirements of Q-learning,the Bellman operator contraction theorem of single-agent is extended to distributed learning,the partial observability theory under multi-agent reinforcement learning is verified for the first time,and its influence on convergence is analyzed.Based on the above theoretical analysis,the reward mechanism of reinforcement learning is modified,and a fully distributed multi-agent reinforcement learning algorithm based on reward recorder is proposed.The decision domain of the learning process is limited to strict constraints by the distributed linear matrix feedback method,and the performance of the algorithm is verified under the generalized distributed optimization model with constraints.2.Research on fully distributed multi-agent reinforcement learning algorithm based on actor-critic.Aiming at the convergence requirement of critic-actor learning,based on partial observability theory,the deployment requirement of distributed critics is analyzed.Based on the above theoretical analysis,the agent’s global observation field is expanded,while ensuring the privacy requirements of the multi-agent system.A multi-agent proximal strategy optimization algorithm based on distributed observers is proposed,and an improved distributed Fibonacci mapping is given to limit the action space of the learning process within strict global constraints.3.Research on cooperative non-convex economic dispatch problem considering target uncertainty.In order to extend the fully distributed off-policy algorithm to deep learning,based on stochastic process convergence analysis,the process condition of multi-agent reinforcement learning convergence is relaxed to the final value condition,and a fully distributed multi-agent deep reinforcement learning based on target value competition is proposed.Aiming at the collaborative economic dispatch problem considering non-convex and uncertain objectives,the influence of optimization target uncertainty is eliminated by introducing a reward network,and a framework for offline learning and online execution is constructed in combination with the generalization ability of deep reinforcement learning.The real-time response to demand-side uncertainty is realized by learning in stochastic states.4.Research on coordinated management of multi-area integrated energy systems considering energy conversion and renewable energy.Aiming at the multi-area electricity-heat-wind integrated energy system,a hierarchical decision-making framework is proposed,which assigns different priorities to the intra-area thermal energy dispatch and the inter-area electric energy dispatch,so as to improve the utilization rate of new energy.Based on multi-agent deep reinforcement learning,a distributed energy management strategy is proposed,and each traditional generator unit,CHP unit,and wind turbine group is defined as an independent decision-making agent.Through learning in stochastic states,a fast response to demand-side uncertainty and renewable energy uncertainty is achieved while minimizing operating costs.