Evidencing responsibility for use of AI in safety critical systems

In recent years there has been much dialogue surrounding concepts of "responsible AI" in areas such as ethics, fairness and risk of existential harm from generative AI. However, this dialogue is rarely targeted at AI-based Safety Critical Systems (AI-SCS), which have many unique regulatory and disciplinary challenges compared to other domains. Safety engineers are being increasingly required through regulation to evidence responsible use of AI, but the discipline lacks the conceptual clarity or methodology to do so. This multi-disciplinary paper uses philosophical models of responsibility (moral, causal, role and legal) to provide clarity for the discipline of safety engineering. We consider AI-SCS challenges including causal responsibility gaps, the risk of a human in the loop being unfairly blamed after an AI-SCS accident, and the problem of "many hands" hiding responsibility during development. We propose presenting evidence of responsible AI use via responsibility models, suitable for safety engineers to present as evidence as part of a system safety justification. We illustrate the application of our approach with two different contrasting examples. The first is a retrospective accident analysis of the death of a pedestrian in Tempe, Arizona 2018 involving an autonomous vehicle. The second is a predictive example for an AI-based clinical decision support tool. We show that by using our approach we can uncover residual risk and improve safety, allocating tasks to the most appropriate responsible actors. We conclude that using our models can support safety engineers in demonstrating responsible use of AI. We also identify complex issues around moral answerability and causal contribution for safety tasks.