Study On Air Conditioning Load Flexibility Regulation Strategy For Zero Energy Houses Based On Reinforcement Learning

In recent years,due to significant changes in the demand structure of domestic load,building energy consumption has been increasing and showing an upward trend year by year.During the summer peak load period,the air conditioning load currently accounts for 30% to 40% of the peak load,and the huge air conditioning load has become an important reason for the continuous increase in building load,and also poses a great threat to the economic and safe operation of the power grid.Therefore,optimizing the operation of air conditioning systems is the main solution for building energy conservation and the economic and safe operation of the power grid.However,most of the traditional control strategies currently adopted by most buildings in real life,including rule-based control or simple on/off of devices,cannot achieve ideal control effects.Reinforcement learning control of building energy systems without models requires a large amount of training data,and the learning efficiency is low,which brings difficulties for the practical application of the model.To solve the above problems,this paper proposes a hybrid reinforcement learning framework based on a grey box model(reduced-order thermodynamic model),which uses real-time monitoring data with short periods to optimize the indoor thermal comfort and energy performance of zero-energy homes(ZEH).The design of the reward function consists of space heating cost and indoor thermal comfort,and the proposed intelligent agent interacts with the grey box model in an unknown environment to obtain reward values as experience for continuously updating the parameters of the decision network,thus learning the optimal decision to achieve maximum cumulative reward or to reach specific goals.To verify the effectiveness of the reinforcement learning algorithm based on the grey box model,this study used four indicators,namely,indoor temperature,weekly accumulated cost,air conditioning power,and on-site photovoltaic consumption ratio,to evaluate the effects of the trained reinforcement learning algorithms,including DQN discrete control strategy,TD3 continuous control strategy,and traditional PI control method.The test results show that compared with traditional control strategies,reinforcement learning algorithms can simultaneously reduce energy costs while maintaining indoor thermal comfort and accommodating more photovoltaic power generation.In the interaction with the grey-box model,the DQN discrete control algorithm can more accurately learn the optimal temperature control strategy,while the TD3 control strategy can better adjust the operation of the air conditioning system to achieve more efficient photovoltaic energy consumption.In the test week with low outdoor ambient temperature,the energy system controlled by the DQN algorithm spent a total of 1,261 Yen for the week,saving 7 Yen compared with the TD3 control strategy and 1,295 Yen compared with the PI control strategy.In terms of indoor thermal comfort,the energy system controlled by the TD3 algorithm had an average indoor temperature close to the set indoor expected average temperature value,which was 22.0 ℃,while the average indoor temperature under the DQN control strategy was 21.4℃,and under the PI control strategy it was 23.5℃.In terms of on-site photovoltaic consumption,the energy system controlled by the TD3 algorithm had a cumulative on-site consumption of 40,186 W for the week,which was 14.7% higher than the DQN control strategy and 32.4% higher than the PI control strategy.In the test week with high outdoor ambient temperature,the energy system controlled by the TD3 control algorithm spent a total of 594 Yen for the week,saving 16 Yen compared with the DQN control strategy and 691 Yen compared with the PI control strategy.In terms of controlling indoor temperature,the average indoor temperature under the DQN control strategy was 21.8℃,while the average indoor temperature under the TD3 control strategy and the PI control strategy was 23.1℃ and23.7℃,respectively,which were closer to the set indoor expected average temperature value.In terms of on-site photovoltaic consumption,the TD3 control strategy had a cumulative on-site consumption of 45,288 W for the week,which was 18.7% higher than the DQN control strategy and 83.4% higher than the PI control strategy.In general,the DQN discrete control algorithm can learn the optimal temperature control strategy more accurately during the interaction with the gray-box model,while the TD3 control strategy can better adjust the operation of the air conditioning system to achieve more efficient photovoltaic energy consumption.These results further demonstrate the effectiveness and practicality of reinforcement learning algorithms based on gray-box models,providing new ideas and technical support for the intelligent application of energy management systems.