Safety engineering, role responsibility and lessons from the Uber ATG Tempe Accident

Safety critical autonomous systems (SCAS) require a safety assurance case (SAC) to justify why they are considered acceptably safe to use, despite the residual risk associated with their operation. Reducing risk is an overarching principle of all safety critical systems development and operation. The SAC should demonstrate that the risk is tolerable and has been reduced as far as possible, through robust design and operational controls. As a SCAS may not have an operator, safety engineers have a more direct responsibility for operational decisions. Following an accident it may be useful to understand which engineering decisions causally contributed to it, and roles responsible for those decisions. This paper contains a review of how different senses of responsibility (role, moral, legal and causal) apply to SCAS engineering and operation. We use this to illustrate how considering role responsibility can help support a defensible SAC, and potentially improve system safety practice. Our findings are illustrated with an analysis the Uber/Tempe Arizona fatal collision accident report. We found that existing safety practice may not identify all role responsibilities in a way that supports causal safety analysis. This paper is intended for the whole TAS community, but with an emphasis on safety professionals.