Reinforcement Learning Based Energy Management Strategy Of Hybrid Ship

Today,the world economy is booming,the marine transportation industry is growing,and the types and number of ships are increasing,which result in the continuous development of water pollution and noise.The emergence of hybrid ships can solve the problems of environmental pollution and energy consumption to a great extent.How to coordinate the power flow of various power sources of hybrid ships and make all parts of the ship work safely and efficiently has become an urgent problem to be solved in the field of ships.Therefore,the energy management strategy of hybrid ships has become the focus of current research.In this paper,the inland river ship taking 690 passengers named ‘Meiwei Kaiyue’is taken as the target ship.Considering the hybrid system structure composed of diesel generator set and supercapacitor,the energy management strategies based on rules,Pontryagin’s minimum principle and reinforcement learning algorithm are proposed,and the effects of each energy management strategy are simulated and analyzed.The specific research contents include:(1)Using the method of combining theoretical analysis and experimental data modeling,the mathematical models of various components in the hybrid power system are established,including diesel generator set model,supercapacitor model and load model.Finally,by analyzing the energy flow of the hybrid system,the system power balance equation is established;(2)A rule-based energy management strategy is proposed.Through the division of the working mode of the hybrid system and the setting of the state threshold,the control rules of the hybrid ship energy management strategy are determined.The simulation analysis shows that the rule-based energy management strategy can give full play to the characteristics of supercapacitor as power energy storage element.Under the control of this strategy,the average specific fuel consumption of diesel generator prime mover in the whole navigation condition is 243.4 g/(k W·h),and the total fuel consumption is 302.3 kg;(3)Based on the Pontryagin’s minimum principle,the hybrid power management model is designed and the energy management strategy is proposed.The simulation results show that the energy management strategy based on Pontryagin’s minimum principle can optimize the working state of marine diesel engine prime mover and make it work at the point with low specific fuel consumption.Under the control of this strategy,the average specific fuel consumption of diesel generator prime mover in the whole navigation condition is 207.2 g/(k W?h),and the total fuel consumption is 204.1kg;(4)Based on the theory of reinforcement learning,the state space,action space and reward function in the energy management model of hybrid ship are constructed.Based on two classical algorithms,deep Q network and deep deterministic policy gradient,energy management strategies are designed.The simulation results show that the energy management strategy based on deep Q network algorithm and deep deterministic policy gradient algorithm can better optimize the working point of diesel generator set prime mover.Under the control of the two strategies,the average specific fuel consumption of diesel generator prime mover in the whole navigation condition is229.2 g/(k W·h)and 215.6 g/(k W?h)respectively,and the total fuel consumption is257.3 kg and 215.5 kg respectively.Through the comparative analysis of three energy management strategies,it is found that the control rules of rule-based energy management strategy are simple,but the optimization effect is limited.The optimization effect of energy management strategy based on Pontryagin’s minimum principle is the best,but it needs to know the relevant information of the whole working condition in advance,which is difficult to realize engineering application.The energy management strategy based on reinforcement learning has a certain self-learning ability,which better balances the optimization effect and the possibility of practical application.The research of this paper can provide technical support for the stable,safe and efficient operation of hybrid ship power system.