Dynamic safety cases for frontier AI

AI Governance Taskforce

Autumn 2024

Frontier artificial intelligence (AI) systems present both benefits and risks to society. Safety cases - structured arguments supported by evidence - are one way to help ensure the safe development and deployment of these systems. Yet the evolving nature of AI capabilities, as well as changes in the operational environment and understanding of risk, necessitates mechanisms for continuously updating these safety cases. Typically, in other sectors, safety cases are produced pre-deployment and do not require frequent updates post-deployment, which can be a manual, costly process. This paper proposes a Dynamic Safety Case Management System (DSCMS) to support both the initial creation of a safety case and its systematic, semi-automated revision over time. Drawing on methods developed in the autonomous vehicles (AV) sector - state-of-the-art Checkable Safety Arguments (CSA) combined with Safety Performance Indicators (SPIs) recommended by UL 4600, a DSCMS helps developers maintain alignment between system safety claims and the latest system state. We demonstrate this approach on a safety case template for offensive cyber capabilities and suggest ways it can be integrated into governance structures for safety-critical decision-making. While the correctness of the initial safety argument remains paramount - particularly for high-severity risks - a DSCMS provides a framework for adapting to new insights and strengthening incident response. We outline challenges and further work towards development and implementation of this approach as part of continuous safety assurance of frontier AI systems.


Read the AI Policy Bulletin article.

Alumni

Meet the authors

(Research Team Lead)

Ben R Smith

Ben is Programme Lead for the AI Governance Taskforce. Before transitioning to AI Governance, he led a team working on electric propulsion in the aerospace industry, an was previously an engineer in the McLaren F1 team.

Carmen (Cârlan) Noll

Francesca Gomez

Francesca is the founder of Wiser Human, an AI safety and governance organisation working to make advanced AI systems more controllable and governable in practice. As Research Practice Lead for the AI Governance Taskforce, Francesca works with the Taskforce Lead to develop our research management systems and support our Research Team Leaders. Her background spans artificial intelligence, human-centred computing, and operational risk across the financial and technology sectors. Alongside her role at Arcadia, she is currently focused on designing and testing controls for AI coding agents to preserve human oversight as they become more capable, and on developing ways to detect when that oversight is becoming strained or ineffective.

Ketana Krishna

René King

Peter Gebauer

Alumni

Meet the authors

Ben R Smith

(Research Team Lead)

Ben is Programme Lead for the AI Governance Taskforce. Before transitioning to AI Governance, he led a team working on electric propulsion in the aerospace industry, an was previously an engineer in the McLaren F1 team.

Carmen (Cârlan) Noll

Francesca Gomez

Francesca is the founder of Wiser Human, an AI safety and governance organisation working to make advanced AI systems more controllable and governable in practice. As Research Practice Lead for the AI Governance Taskforce, Francesca works with the Taskforce Lead to develop our research management systems and support our Research Team Leaders. Her background spans artificial intelligence, human-centred computing, and operational risk across the financial and technology sectors. Alongside her role at Arcadia, she is currently focused on designing and testing controls for AI coding agents to preserve human oversight as they become more capable, and on developing ways to detect when that oversight is becoming strained or ineffective.

Ketana Krishna

René King

Peter Gebauer

Programme

AI Governance Taskforce

The AI Governance Taskforce is a career development programme for experienced professionals looking to transition careers into AI governance, focussed on reducing risks from advanced AI.
Participants work around existing commitments during our 12 week, remote, part-time cohorts, producing policy research in teams of 4, led by our Research Team Lead staff in partnership with recognised experts in the field. Teams write an academic-style paper and accompanying blog post to build knowledge, skills and work portfolios.