Assurance of AI and autonomous technology in complex environments:a human-centered perspective

The development of AI and autonomous technologies is increasing rapidly across multiple sectors. The safety assurance of such systems often takes a narrow and technology-driven perspective, focusing on technical criteria such as overall performance metrics. However, these systems operate within broader human, technical, and organisational contexts. For example, an autonomous ship exists with-in a maritime system involving other functions and services including, among others, port and docking services, maintenance services, navigational services etc. Without consideration of these broader contexts, hazardous scenarios can arise, which may result from dysfunctional interactions between the autonomous functions and other elements of the maritime system. Effective safety assurance re-quires, therefore, consideration of the interactions between different elements of the work system, where the AI or autonomous system is just one such element. In this paper we put forward human-centered reflections based on an analysis of 22 demonstrator projects from diverse application domains.