A mortality prediction model lost 9 to 39 percent specificity after retraining once at a fixed sensitivity of 90 percent
By Elana Gotkine HealthDay Reporter
TUESDAY, Oct. 10, 2023 (HealthDay News) — In a simulated intensive care unit setting, there is no universally effective model-updating approach that maintains model performance, according to a study published online Oct. 10 in Annals of Internal Medicine.
Using data for 130,000 critical care admissions across two health systems, Akhil Vaid, M.D., from the Icahn School of Medicine at Mount Sinai in New York City, and colleagues estimated changes in predictive model performance with the use of three common scenarios: model retraining, sequential implementation of one model after another, and intervening in response to a model when models are implemented simultaneously.
The researchers found that in scenario 1, a mortality prediction model lost 9 to 39 percent specificity after retraining once at fixed 90 percent sensitivity, and in scenario 2, a mortality prediction model lost 8 to 15 percent specificity when created after implementation of an acute kidney injury (AKI) prediction model. Models for AKI and mortality prediction implemented simultaneously in scenario 3 led to reduced effective accuracy of the other by 1 to 28 percent.
“We found that implementation of predictive models into clinical workflows has the potential to reduce adverse outcomes,” the authors write. “However, because these changes are subsequently captured within the electronic health record, this reduction also disrupts the relationship between the severity of initial presentation and the likelihood of an adverse outcome that predictive models learn to establish.”
Abstract/Full Text (subscription or payment may be required)
Editorial (subscription or payment may be required)
Copyright © 2023 HealthDay. All rights reserved.
