From development to deployment: actionable AI models which accurately predict patient risk in Chronic Obstructive Pulmonary Disease

Chronic obstructive pulmonary disease (COPD) is a common respiratory disorder and is the third leading cause of death worldwide. Deploying validated, accurate, fair and explainable machine learning models can transform COPD care. We deployed the first live AI risk prediction model in the UK for COPD patients in 2023 and have since worked on deploying 3 additional models. We have worked with NHS clinicians to design an application for viewing model scores and identifying high-risk patients.

Model development was undertaken on de-identified data within Greater Glasgow & Clyde NHS Trust, with ethics approval. We have currently deployed two models: 12-month mortality risk and a 90-day hospital readmission model and are working on deploying additional models. These models use routinely collected electronic health record (EHR) data – demographics, hospital admission, prescribing and labs from more than 28000 patients. Model training and cross-validation involved 80% of the patient cohort, with a matched 20% reserved for the holdout test cohort.

The mortality and readmission models achieved ROC-AUC scores of 0.85 and 0.73 and PR-AUC scores of 0.46 and 0.63 on the respective holdout test cohort. These models were deemed useful for stratifying and prioritising high-risk patients during clinical advisory review. Global explainability using SHAP was used to identify general patterns and local explainability was used to provide patient-specific insights for personalised interventions.

Our deployed models have identified high-risk patients in a clinical setting. Clinicians have conducted a variety of interventions based on the model output, such as closer monitoring of high-risk patients, reviewing patient inhaler-technique, and adjusting medication. In this talk, I will go through some of the challenges we have encountered developing models and getting them into production. I will focus on scalability, continuous model monitoring, regulatory constraints and challenges associated with integrating AI into clinical workflows.

Technical Level: Technical practitioner

Supported by