Jesper Weile

The Framing of machine learning risk prediction models illustrated by evaluation of sepsis in general wards

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review


Problem framing is critical to developing risk prediction models because all subsequent development work and evaluation takes place within the context of how a problem has been framed and explicit documentation of framing choices makes it easier to compare evaluation metrics between published studies. In this work, we introduce the basic concepts of framing, including prediction windows, observation windows, window shifts and event-triggers for a prediction that strongly affects the risk of clinician fatigue caused by false positives. Building on this, we apply four different framing structures to the same generic dataset, using a sepsis risk prediction model as an example, and evaluate how framing affects model performance and learning. Our results show that an apparently good model with strong evaluation results in both discrimination and calibration is not necessarily clinically usable. Therefore, it is important to assess the results of objective evaluations within the context of more subjective evaluations of how a model is framed.

Tidsskriftnpj Digital Medicine
StatusUdgivet - nov. 2021

Bibliografisk note

Funding Information:
We acknowledge the steering committee for CROSS-TRACKS for access to the data. For help with acquisition, modeling, and validation of the data extraction pipelines, we thank Per Dahl Rasmussen, Rasmus Holm Laursen, Anne Olsvig Boilesen, Emil Møller Bartels, and Christian Bang. We also thank the rest of the Enversion team for their support. This work was funded by the Innovation Fund Denmark (case number 8053-00076B).

Publisher Copyright:
© 2021, The Author(s).

Se relationer på Aarhus Universitet Citationsformater

ID: 226792045