Ryan, Philippa Mary orcid.org/0000-0003-1307-5207, Porter, Zoe, Fearnley, Laura Christina Anne et al. (4 more authors) (2025) Evidencing responsibility for use of AI in safety critical systems. [Preprint]
Abstract
In recent years there has been much dialogue surrounding concepts of "responsible AI" in areas such as ethics, fairness and risk of existential harm from generative AI. However, this dialogue is rarely targeted at AI-based Safety Critical Systems (AI-SCS), which have many unique regulatory and disciplinary challenges compared to other domains. Safety engineers are being increasingly required through regulation to evidence responsible use of AI, but the discipline lacks the conceptual clarity or methodology to do so. This multi-disciplinary paper uses philosophical models of responsibility (moral, causal, role and legal) to provide clarity for the discipline of safety engineering. We consider AI-SCS challenges including causal responsibility gaps, the risk of a human in the loop being unfairly blamed after an AI-SCS accident, and the problem of "many hands" hiding responsibility during development. We propose presenting evidence of responsible AI use via responsibility models, suitable for safety engineers to present as evidence as part of a system safety justification. We illustrate the application of our approach with two different contrasting examples. The first is a retrospective accident analysis of the death of a pedestrian in Tempe, Arizona 2018 involving an autonomous vehicle. The second is a predictive example for an AI-based clinical decision support tool. We show that by using our approach we can uncover residual risk and improve safety, allocating tasks to the most appropriate responsible actors. We conclude that using our models can support safety engineers in demonstrating responsible use of AI. We also identify complex issues around moral answerability and causal contribution for safety tasks.
Metadata
Item Type: | Preprint |
---|---|
Authors/Creators: |
|
Dates: |
|
Institution: | The University of York |
Academic Units: | The University of York > Faculty of Sciences (York) > Computer Science (York) The University of York > Faculty of Social Sciences (York) > The York Law School |
Depositing User: | Pure (York) |
Date Deposited: | 05 Mar 2025 09:20 |
Last Modified: | 05 Mar 2025 09:20 |
Status: | Published |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:224048 |
Download
Filename: 7DE15A22-F2C7-11EF-97F9-F05E47712E31.pdf
Description: 7DE15A22-F2C7-11EF-97F9-F05E47712E31
Licence: CC-BY 2.5
