Exploring Safe Reinforcement Learning Using Safety Shields Derived With System-Theoretic Process Analysis: A Case-Study on a Cruise Ship Hotel System

The cruise ship industry is under increasing pressure to reduce greenhouse gas emissions, as international regulations define ambitious requirements and goals for modern cruise ships. One of the most significant consumers of energy onboard cruise ships are their Heating Ventilation and Air Conditioning (HVAC) systems. However, the energy optimization of HVAC systems is challenging, as they are impacted by a number of uncontrolled variables, such as changing weather conditions, passenger behavior, and the demands of other significant energy consumers, such as propulsion systems. Reinforcement Learning (RL) is often used to tackle such complex optimization tasks, however concerns over ensuring the safety of RL optimized systems hinders its adoption in industry, especially in the context of safety-critical systems. This paper presents the initial findings of applying a novel approach to ensure safety in RL: a safety shield developed utilizing a novel hazard analysis method, System-Theoretic Process Analysis. In this work the safety shield is used to both train the RL agent as well as block unsafe behavior in operation. Preliminary findings suggest that blocking unsafe behavior during training hinders the ability to learn a safe RL policy, however, when used in testing the approach is capable of significantly reducing the number of safety violations.