Transformer Model Based Soft Actor-Critic Learning for HEMS

Ulrich Ludolfinger, Vedran S. Perić, Thomas Hamacher, Sascha Hauke, Maren Martens

Research output: Contribution to book/anthology/report/proceedingArticle in proceedingsResearchpeer-review


The transition to weather dependent renewable energy generators requires the electric loads to be adjusted to generation. This is made possible by demand response programs and home energy management systems. However, practically easy to use rule-based control systems often miss many optimization potentials. Self-learning alternatives employing reinforcement learning often ignore the partial observability of the building control problem and consequently neglect the importance of the observation history. Adaptive control systems that do consider that history often rely on policies that suffer from catastrophic forgetting, which makes them unable to fully grasp long histories. As an alternative, we present a new reinforcement learning method for autonomous building energy management control based on the soft actor-critic method and the transformer deep neural network architecture. For the control of a heat pump and an the inlet port of a thermal storage, under consideration of photovoltaic generations and dynamic electricity prices, we formulate the problem as partially observable and use the history of observations to determine the control signals. We show, based on a validated building simulation, that our method outperforms rule-based as well as reinforcement learning methods that use multi layer perceptrons or recurrent neural networks as policy.

Original languageEnglish
Title of host publication2023 International Conference on Power System Technology (PowerCon)
Publication date2023
ISBN (Electronic)979-8-3503-0022-2, 979-8-3503-0023-9
Publication statusPublished - 2023
Externally publishedYes
Event2023 International Conference on Power System Technology, PowerCon 2023 - Jinan, China
Duration: 21 Sept 202322 Sept 2023


Conference2023 International Conference on Power System Technology, PowerCon 2023
SponsorChinese Society for Electrical Engineering, IEEE Power and Energy Society, State Grid Corporation of China


  • demand response
  • home energy management
  • machine learning
  • reinforcement learning
  • soft actor-critic
  • transformer model


Dive into the research topics of 'Transformer Model Based Soft Actor-Critic Learning for HEMS'. Together they form a unique fingerprint.

Cite this