Optimal Management Of Battery Energy Storage System Based On Reinforcement Learning

The generated power of distributed generation units, represented by wind power, photovoltaic power, is difficult to ensure stable, changes with time and climate. The instable power may cause the frequency and voltage instability, and then cause power outages. To solve this problem, we introduced into the energy storage device in the system with distributed generation units. The battery may be in a loss of power state or overcharge state for a long time with the imbalance in supply and demand, may shorten the battery life, and then increases maintenance costs. So there is important practical significance in selecting the appropriate battery control strategy.The thesis considers an energy storage system which consists of distributed generation units, storage device, loads and some intelligent control devices. It enables energy flow from the storage device to the grid. An amount of balancing energy is procured to meet the load demand when there is a deficit in power generation. The excessive distributed generation power of storage device can either be sold to the grid or be used to provide frequency regulation service. The power of distributed generation units, load demand, electricity price and the frequency regulation price are independent of each other. Each of the four stochastic processes is modeled as a Markov process to reflect the dynamic characteristics. In practical application, the battery turns to empty or full, sojourn time of the system state is not exponential distribution. So the optimal control problem of deciding when to sell power, buy power or provide regulation service is formulated as a Semi-Markov decision process. The model-based Sarsa algorithm is used to select the optimal policy in order to maximize the long-term rewards on the basis of meeting the load demand.With the development of the electric car industry, vehicle-to-grid (V2G) is becoming a hot topic. The thesis considers adding distributed generation device into the V2G system. An amount of balancing energy is procured when there is a deficit in power generation. The excessive distributed generation power is sold to the grid. The energy of the vehicle battery can either be sold to the grid or be used to provide frequency regulation service based on the pricing information. We assume the system receives the generation power and pricing information a few minutes prior to the beginning of each hour. The optimal control problem, of this system is formulated as a dynamic programming process. The policy iteration is used to get the optimal policy in order to maximize the long-term rewards on the basis of meeting the vehicle’s power demand.