Development, external validation and integration into clinical workflow of machine learning models to support pre-operative assessment in the UK

Introduction Demand for surgical treatment is growing and patient complexity is increasing. The NHS England standard contract now requires that pre-operative services risk stratify and optimise patients awaiting surgery. However, current pre-operative workflows (whether electronic or paper-based) remain based primarily on resource-intensive manual tasks. Lack of real-time data transfer has been identified as a key limitation to reducing the surgical backlog.

Methods We developed certified electronic linkages between a live pre-operative assessment system (Smart PreOp, Aire Logic Ltd, Leeds, UK) and the GP Connect system from NHS England to retrieve clinical data directly from general practitioner records into pre-operative questionnaires. We developed machine learning models to categorise patients into lower- and higher-risk cohorts based on their predicted ASA physical status (1 or 2 vs. 3–5) and 30-day postoperative mortality risk. In contrast with previous prediction modelling studies, we constrained variable selection from the outset to variables that are available electronically in real time for all UK surgical patients regardless of where they present (the proposed procedure, demographics and medications lists).

Results The development and external validation cohorts consisted of 110,732 and 67,878 patients, respectively, from two NHS Trusts using different electronic record systems. In external validation, at decision threshold 0.2, the ASA physical status prediction model had recall 0.69 and precision 0.95 for identifying lower-risk (ASA physical status 1 or 2) patients. The mortality prediction model discriminated well in external validation but was poorly calibrated, lending support to the existing literature showing that hospital-specific modelling improves mortality risk prediction. The technical architecture of the Smart PreOp system facilitates such hospital-specific modelling and periodic model updates.

Discussion We conclude that conducting modelling together with systems development can yield accurate prediction models that may be implemented directly into electronic health records. A prospective study of clinical impact and acceptability is warranted.