STAMP/STPA informed characterization of Factors Leading to Loss of Control in AI Systems

Many capabilities of frontier AI, such as agency, situational awareness and evasion of oversight could contribute to loss of control. The range of concerns is wide, spanning current day risks to future existential risks, and scenarios ranging from gradual disempowerment to rapid AI self-exfiltration. Given this variety, we set out to explore the production of a structured framework for discussing and characterizing loss of control. We also set the objective that our framework should be of practical assistance to those responsible for the safe operation of AI-containing socio-technical systems, when identifying causal factors leading to loss of control. System-Theoretic Accident Model and Processes (STAMP) and its associated hazard analysis technique, System-Theoretic Process Analysis (STPA) [Leveson, 2012], holds promise in helping to address these two objectives.

In our work we demonstrate how an AI loss of control characterization framework based around STAMP/STPA can be used to identify and characterize some types of AI loss of control. We show how certain AI characteristics may be traced through causal pathways to a loss of control event, and we show an approach through which guidance in hazard identification can help those responsible for ensuring the safety of AI-based systems. Our exploration focused on a simple, though important and fundamental, control system archetype. As such, there are a number of important more complex AI loss of control scenarios and system archetypes which would need to be explored in any future work. These include, for example, systems with multiple AI agents, AI systems that recursively self-improve, diffuse systems where no single controller exists, and systems with hybrid human-AI controllers.

[Leveson, 2012] ‘Engineering a safer world: Systems thinking applied to safety’, The MIT press

Expert Partner: Richard Mallah (CARMA - Centre for AI Risk Management and Alignment)