Creating an effective methodology for end-user engagement in AI auditing

A methodology explains the object of an AI-audit. This object has three loci: identifying significant events (harms or risks), governance (model is behaving as expected), and assurance (trust). The methodology in this paper is being developed as part of the PHAWM project (The Participatory Harm Auditing Workbenches and Methodologies project can be found at https://phawm.org), which seeks to design a workbench that supports inclusive, participant-led auditing of AI application across a range of domains. Project participants range from health service users, parents of school-aged children, to museum professionals and librarians. The project addresses a key gap in existing approaches: the absence of human-centred infrastructures that empower end-users to identify events (An event refers to an occurrence triggered by an AI application that may affect entities and has associated metrics. Each event can be assessed for likelihood, magnitude, and positive or negative valence. We avoid the term harm in our methodology due to its subjectivity, although we acknowledge its common use, including in our own project title, within AI auditing discourse), understand system behavior and participate meaningfully in audit processes.