Safety assurance of Machine Learning for autonomous systems

Machine Learning (ML) components are increasingly incorporated into systems, with different degrees of autonomy, where model performance is reported as meeting, or exceeding, the capabilities of human experts. This promises to transform products and services, in diverse domains such as healthcare, transport and manufacturing, to better serve underrepresented groups, reduce costs, and increase delivery effectiveness, especially where expert resources are scarce. The greatest potential for transformative impact lies in the development of autonomous systems for safety-critical applications where their acceptance, and subsequent deployment, is reliant on establishing justified confidence in system safety. Creating a compelling safety case for ML is challenging however, particularly since the ML development lifecycle is significantly different to that employed for traditional software systems. Typically ML development involves replacing detailed software specifications with representative data sets from which models of behaviour is learnt. Indeed, ML’s strength lies in tackling problems which are challenging for traditional software development practices. This shift in development practices introduces challenges to established assurance processes which are crucial to developing the compelling safety case required for regulation and societal acceptance. In this paper we introduce the first methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). The AMLAS process describes how to systematically and attractively integrate safety assurance into the development of ML components and how to generate the evidence base for explicitly justifying the acceptable safety of these components when integrated into autonomous system applications. We describe the use of AMLAS by considering how a safety case may be constructed for an object detector for use in the perception pipeline of an autonomous driving application. We further discuss how AMLAS has been applied in several domains including healthcare, automotive and aerospace as well as supporting policy and industry guidance for defence, healthcare and automotive.