Reinforcement learning in hybrid electric vehicles (HEVs) / electric vehicles (EVs)

The conventional internal combustion engine (ICE)-powered vehicles have contributed significantly to the development of modern society. However, they have also brought about large amounts of fuel consumption and pollution emissions due to the increasing number of vehicles in use around the world. Electric vehicles (EVs) and hybrid electric vehicles (HEVs) have been developed to improve the fuel economy and reduce the pollution emissions.;This thesis first introduces basic components of EV and HEV and methods for the EV/HEV energy management. After an accurate and detailed modeling of the HEV, this thesis provides two control strategies for the HEV energy management to improve the fuel economy. Different from some previous literature work that rely on a priori knowledge of the driving profiles, the proposed control strategies, namely, a Markov decision process based strategy and a reinforcement learning based strategy, only need stochastic knowledge of the driving profiles or do not rely on any prior knowledge of the driving profiles. In particular, the reinforcement learning based control strategy can be model-free, which enables one to (partially) avoid reliance on complex HEV modeling while coping with driver specific behaviors.;The state-of-health (SoH) of the battery pack is degrading with the operation of an HEV. The battery pack will reach its end-of-life when it loses 20% or 30% of its nominal capacity. At the same time, the battery pack replacement results in additional operational cost for an HEV. Therefore, this thesis investigates the energy management problem in HEVs focusing on the minimization of the operating cost of an HEV, including both fuel and battery replacement cost. A nested learning framework is proposed, in which the inner-loop learning process is the key to minimization of the fuel usage whereas the outer-loop learning process is critical to minimization of the amortized battery replacement cost.;On the other hand, auxiliary systems of HEVs/EVs, comprised of lighting, air conditioning (or more generally, heating, ventilation, and air conditioning), and other battery-powered systems such as GPS, may account for 10% - 30% of the overall fuel consumption for an ordinary (fuel-based) vehicle. For HEVs and EVs, it is projected that auxiliary systems will take a larger portion of the overall energy consumption, partly because heating of an ordinary vehicle can be partially achieved by the heated internal combustion engine. Hence, in this thesis, the control of HEV powertain and auxiliary systems are jointly considered for the minimal operational cost. We minimize fuel cost induced both by propelling the vehicle and by the auxiliary systems, and meanwhile maximize a total utility function (representing the degree of desirability) of the auxiliary systems. To further enhance the effectiveness of the RL framework, the prediction of future driving profile characteristics is incorporated.;An EV with onboard PV electrical energy generation system (PV system) is beneficial since PV cells can charge the EV battery pack when the EV is running and parking to mitigate the power demand from the grid. This thesis aims at maximizing the output power of a vehicular PV system with the string charger architecture taking into account the non-uniform distribution of solar irradiance levels on different vehicle surface areas. This work is based on the dynamic PV array reconfiguration architecture from previous work with the accommodation of the rapidly changing solar irradiance in the onboard scenario. Most importantly, this work differs from previous dynamic PV array reconfiguration work in that an event-driven and a sensorless PV array reconfiguration framework are proposed.;The concept of vehicle-to-grid (V2G) was developed to make use of the electrical energy storage ability of EV/HEV batteries for frequency regulation, load balancing, etc. This thesis also presents the work on the smart grid optimal pricing policy problem, in which the aggregator maximizes its profit by designing a real-time pricing policy while taking into account the behaviors of both residential users and EV/HEV users. The aggregator pre-announces a pricing policy for an entire billing period, then in each time interval of the billing period, the electricity users (both residential and EV/PHEV users) try to maximize their own utility functions based on the pricing model in the current time interval and the awareness of the other users' behaviors. We use a dynamic programming algorithm to derive the optimal real-time pricing policy for maximizing the aggragator's overall profit, based on backward induction.