Real-time Predictive Energy Management Strategy Of A PHEV Based On Double Delay Q-Learning

Plug-in Hybrid Electric Vehicle(PHEV)is a bridge between conventional fuel vehicles and electric vehicles.Its excellent energy efficiency makes it a promising solution for alleviating energy crises and air pollution.Energy Management Strategy(EMS)is one of the key technologies for PHEV’s energy consumption.In order to further explore the fuel-saving potential of PHEV,this paper takes power-split PHEV as the research object,and focuses on a real-time energy management strategy based on Double Delay Q-learning(2DQL)and Model Predictive Control(MPC).The main research of this paper is as follows:1)A detailed investigation of the configuration of the power-split PHEV powertrain and the energy flow relationship in each operating mode is presented.The forward simulation model of PHEV was established by modeling the crucial components of PHEV drivetrain on Autonomie platform utilizing Matlab/Simulink software.In the end,based on the internal allocation of PHEV and the features of each operating model,an EMS based on CD/CS algorithm is designed,and the simulation verifies the validity of the model to lay the foundation for the next strategy research.2)For the fuel economy of PHEV,two different optimization algorithms based on Equivalent Consumption Minimization Strategy(ECMS)and Stochastic Dynamic Programming(SDP)are designed for EMS.The simulation results reveal that although the ECMS-based EMS has fine optimization effect,its fuel economy is not as favorable as the SDP-based EMS.Meanwhile,the SDP-based EMS does not require advance information on working conditions and can achieve global sub-optimal fuel economy,which provides a reference foundation for the construction of subsequent intelligent EMS.3)Considering the limitations of rule-based and optimization-based strategies,an EMS based on double-delay Q-learning is designed,which does not rely on the system model.To begin with,the algorithm concept of reinforcement learning is fully explaine,and many reinforcement learning algorithms are categoried according to whether they rely on models or not;Secondly,the traditional Q learning algorithm and its two variants are elaborated;Then,aiming at the problem of overestimation of traditional Q learning and underestimation of double Q learning,combined delayed Q learning and double Q learning,an EMS based on double delayed Q learning was proposed.Through the update of delayed Q learning,the underestimation of Q function occurred in the double delayed Q learning algorithm was avoided.At the same time,the strategy deviation occurred in delayed Q learning was eliminated by using two independent Q functions for each stateaction pair.At last,the result shows that the EMS based on double delay Q algorithm under different test cases has better control effect compared with other strategies,and the power can be distributed more reasonably among PHEV power units to achieve better fuel economy.4)In the framework of model prediction control(MPC),a real-time EMS based on vehicle speed prediction is constructed by combining with double delay Q learning.First of all,with the advantages and disadvantages of various types of vehicle speed prediction,two speed prediction models are set up and their predictions are compared under test working conditions.After analysis,a convolutional neural network-based speed prediction model is finally selected as the prediction model of MPC in this paper.Then,an EMS capable of real-time predictive control is designed based on the double delay Qlearning offline controller obtained above to solve the optimal control problem in the rolling optimization process.By analyzing the fuel economy and computational efficiency of the proposed strategy under different prediction accuracy,it is verified that the EMS based on 2DQL-MPC not only has excellent performance in fuel economy and power constraint processing of power battery,but also has excellent real-time performance.