Reinforcement Learning-Based Energy Management For Hybrid Electric Vehicles

As a promising solution to global warming and air pollution,hybrid electric vehicles(HEVs)are becoming increasingly popular.Energy management strategy plays a crucial role in affecting performance,cost effectiveness,and adapatability of HEVs by controlling and distributing energy among multiple energy storage systems.In order to improve the efficiency performance of improving fuel economy or reducing emissions for a HEV,a highly efficient and real-time energy management strategy is necessary.Owing to the being stochastic of power request and adaptive demand of energy management strategy,reinforcement learning(RL)-based energy management and control is drawing increasing attentions from academia and industry,which is of great scientific value for the development and popularization of hybrid electric vehicles.This dissertation presents a novel reinforcement learning-based energy management structure in real time for power-split hybrid electric vehicles.The main contributions include:First,the control oriented model of a parallel hybrid electric vehicle(HEV)is established,the power split among multiple energy storage systems is formulated as the optimal control problem.We compare Dynamic Programming(DP)and Pontryagin's Minimum Principle(PMP)for the parallel HEV using an Automatic Manual Transmission(AMT).Tuning with the appropriate co-states,the PMP solution is found to be very close to that from DP.The conclusion that the co-states in PMP equal to the partial derivative of the cost function with respect to the corresponding state variable in DP can be applied for benchmarking the reinforcement learning algorithms.The second part of the work builds a control oriented model for a series hybrid tracked vehicle(HTV).The forward simulation model of the HTV is formed in Matlab/Simulink,including the engine-generator model,the battery model,the motor model and vehicle dynamics model.Two reinforcement learning(RL)algorithms named Q_learning and Dyna are analyzed and compared for the series HTV.The iterative criterion of these two algorithms is illuminated in a theoretical perspective.Subsequently,the RL-based energy management strategy is compared with the DP-based energy management strategy.The simulation results indicated that the Q-learning algorithm entailed a lower computational cost(3h)compared with the Dyna algorithm(7h);nevertheless,the fuel consumption of the Q-learning algorithm was 1.7% higher than that of the Dyna algorithm.The Dyna algorithm registered almost the same fuel consumption as the DP-based global optimal solution does.Thirdly,we systematically discuss the optimality,adaptability,and learning ability of the RL technique.The transition probability matrix(TPM)of the power request is formulated based on the Markov chain(MC)theoretically for the HTV.Futuremore,the RL-based energy management strategy is compared with the stochastic dynamic programming(SDP)-based energy management strategy to validate its optimality and demonstrate its adaptability for different driving schedules.The learning ability of the RL-based energy management strategy is clarified by comparing with the new RL-based energy management strategy based on a new TPM.In the fourth part,a RL-based real-time energy management strategy is proposed for the HTV.To use effectively the statistical information of power request online,a MC-based realtime recursive algorithm for learning transition probabilities is derived and validated.The Kullback-Leibler(KL)divergence rate is adopted to determine when the TPM and the optimal control strategy update in real time.RL method is applied to compare quantitatively the effects of different forgetting factors and KL divergence rates on reducing fuel consumption.RL-based energy management strategy is compared with the preliminary and DP-based control strategies and the simulation results indicate the proposed RL-based energy management strategy can significantly improve fuel efficiency and can be applied in real time.In the fifth part of the work,we construct a hardware-in-the-loop(HIL)test to verify the RL-based energy management strategy.The model in Matlab/Simulnik,the Moto Tron controller and the RT-LAB are combined to constitute the test bench.The RL-based energy management strategy is compared with the rule-based strategy to demonstrate its superiorities.We also validate the RL-based energy management strategy in the real-time small HTV.The flowchart of real-time vehicle test and the structure of the vehicle are introduced first.The experiment results based on the RL-based energy management strategy are clarified to validate its instantaneity and effectiveness.