Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning

Martin N. Johansen; Erik T. Parner; Mikkel F. Kragh; Keiichi Kato; Satoshi Ueno; Stefan Palm; Manuel Kernbach; Başak Balaban; İpek Keleş; Anette V. Gabrielsen; Lea H. Iversen; Jørgen Berntsen

doi:10.1007/s10815-023-02871-3

Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning

Martin N. Johansen^*, Erik T. Parner, Mikkel F. Kragh, Keiichi Kato, Satoshi Ueno, Stefan Palm, Manuel Kernbach, Başak Balaban, İpek Keleş, Anette V. Gabrielsen, Lea H. Iversen, Jørgen Berntsen

^*Corresponding author for this work

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

6 Citations (Scopus)

8 Downloads (Pure)

Abstract

Purpose: This article aims to assess how differences in maternal age distributions between IVF clinics affect the performance of an artificial intelligence model for embryo viability prediction and proposes a method to account for such differences. Methods: Using retrospectively collected data from 4805 fresh and frozen single blastocyst transfers of embryos incubated for 5 to 6 days, the discriminative performance was assessed based on fetal heartbeat outcomes. The data was collected from 4 clinics, and the discrimination was measured in terms of the area under ROC curves (AUC) for each clinic. To account for the different age distributions between clinics, a method for age-standardizing the AUCs was developed in which the clinic-specific AUCs were standardized using weights for each embryo according to the relative frequency of the maternal age in the relevant clinic compared to the age distribution in a common reference population. Results: There was substantial variation in the clinic-specific AUCs with estimates ranging from 0.58 to 0.69 before standardization. The age-standardization of the AUCs reduced the between-clinic variance by 16%. Most notably, three of the clinics had quite similar AUCs after standardization, while the last clinic had a markedly lower AUC both with and without standardization. Conclusion: The method of using age-standardization of the AUCs that is proposed in this article mitigates some of the variability between clinics. This enables a comparison of clinic-specific AUCs where the difference in age distributions is accounted for.

Original language	English
Journal	Journal of Assisted Reproduction and Genetics
Volume	40
Issue	9
Pages (from-to)	2129-2137
Number of pages	9
ISSN	1058-0468
DOIs	https://doi.org/10.1007/s10815-023-02871-3
Publication status	Published - Sept 2023

Keywords

Artificial intelligence
Embryo selection
Model performance
Time-lapse
Fertilization in Vitro
Humans
Artificial Intelligence
Blastocyst
Retrospective Studies
Time-Lapse Imaging
Machine Learning

Access to Document

10.1007/s10815-023-02871-3Licence: CC BY

s10815-023-02871-3Final published version, 1.07 MBLicence: CC BY

OpenUrl availability

Full text

Cite this

Johansen, M. N., Parner, E. T., Kragh, M. F., Kato, K., Ueno, S., Palm, S., Kernbach, M., Balaban, B., Keleş, İ., Gabrielsen, A. V., Iversen, L. H., & Berntsen, J. (2023). Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning. Journal of Assisted Reproduction and Genetics, 40(9), 2129-2137. https://doi.org/10.1007/s10815-023-02871-3

@article{f82a6e0b668046aa87a3bab918b84c9a,

title = "Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning",

abstract = "Purpose: This article aims to assess how differences in maternal age distributions between IVF clinics affect the performance of an artificial intelligence model for embryo viability prediction and proposes a method to account for such differences. Methods: Using retrospectively collected data from 4805 fresh and frozen single blastocyst transfers of embryos incubated for 5 to 6 days, the discriminative performance was assessed based on fetal heartbeat outcomes. The data was collected from 4 clinics, and the discrimination was measured in terms of the area under ROC curves (AUC) for each clinic. To account for the different age distributions between clinics, a method for age-standardizing the AUCs was developed in which the clinic-specific AUCs were standardized using weights for each embryo according to the relative frequency of the maternal age in the relevant clinic compared to the age distribution in a common reference population. Results: There was substantial variation in the clinic-specific AUCs with estimates ranging from 0.58 to 0.69 before standardization. The age-standardization of the AUCs reduced the between-clinic variance by 16%. Most notably, three of the clinics had quite similar AUCs after standardization, while the last clinic had a markedly lower AUC both with and without standardization. Conclusion: The method of using age-standardization of the AUCs that is proposed in this article mitigates some of the variability between clinics. This enables a comparison of clinic-specific AUCs where the difference in age distributions is accounted for.",

keywords = "Artificial intelligence, Embryo selection, Model performance, Time-lapse, Fertilization in Vitro, Humans, Artificial Intelligence, Blastocyst, Retrospective Studies, Time-Lapse Imaging, Machine Learning",

author = "Johansen, {Martin N.} and Parner, {Erik T.} and Kragh, {Mikkel F.} and Keiichi Kato and Satoshi Ueno and Stefan Palm and Manuel Kernbach and Ba{\c s}ak Balaban and İpek Kele{\c s} and Gabrielsen, {Anette V.} and Iversen, {Lea H.} and J{\o}rgen Berntsen",

year = "2023",

month = sep,

doi = "10.1007/s10815-023-02871-3",

language = "English",

volume = "40",

pages = "2129--2137",

journal = "Journal of Assisted Reproduction and Genetics",

issn = "1058-0468",

publisher = "Springer",

number = "9",

}

Johansen, MN, Parner, ET, Kragh, MF, Kato, K, Ueno, S, Palm, S, Kernbach, M, Balaban, B, Keleş, İ, Gabrielsen, AV, Iversen, LH & Berntsen, J 2023, 'Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning', Journal of Assisted Reproduction and Genetics, vol. 40, no. 9, pp. 2129-2137. https://doi.org/10.1007/s10815-023-02871-3

Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning. / Johansen, Martin N.; Parner, Erik T.; Kragh, Mikkel F. et al.
In: Journal of Assisted Reproduction and Genetics, Vol. 40, No. 9, 09.2023, p. 2129-2137.

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

TY - JOUR

T1 - Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning

AU - Johansen, Martin N.

AU - Parner, Erik T.

AU - Kragh, Mikkel F.

AU - Kato, Keiichi

AU - Ueno, Satoshi

AU - Palm, Stefan

AU - Kernbach, Manuel

AU - Balaban, Başak

AU - Keleş, İpek

AU - Gabrielsen, Anette V.

AU - Iversen, Lea H.

AU - Berntsen, Jørgen

PY - 2023/9

Y1 - 2023/9

N2 - Purpose: This article aims to assess how differences in maternal age distributions between IVF clinics affect the performance of an artificial intelligence model for embryo viability prediction and proposes a method to account for such differences. Methods: Using retrospectively collected data from 4805 fresh and frozen single blastocyst transfers of embryos incubated for 5 to 6 days, the discriminative performance was assessed based on fetal heartbeat outcomes. The data was collected from 4 clinics, and the discrimination was measured in terms of the area under ROC curves (AUC) for each clinic. To account for the different age distributions between clinics, a method for age-standardizing the AUCs was developed in which the clinic-specific AUCs were standardized using weights for each embryo according to the relative frequency of the maternal age in the relevant clinic compared to the age distribution in a common reference population. Results: There was substantial variation in the clinic-specific AUCs with estimates ranging from 0.58 to 0.69 before standardization. The age-standardization of the AUCs reduced the between-clinic variance by 16%. Most notably, three of the clinics had quite similar AUCs after standardization, while the last clinic had a markedly lower AUC both with and without standardization. Conclusion: The method of using age-standardization of the AUCs that is proposed in this article mitigates some of the variability between clinics. This enables a comparison of clinic-specific AUCs where the difference in age distributions is accounted for.

AB - Purpose: This article aims to assess how differences in maternal age distributions between IVF clinics affect the performance of an artificial intelligence model for embryo viability prediction and proposes a method to account for such differences. Methods: Using retrospectively collected data from 4805 fresh and frozen single blastocyst transfers of embryos incubated for 5 to 6 days, the discriminative performance was assessed based on fetal heartbeat outcomes. The data was collected from 4 clinics, and the discrimination was measured in terms of the area under ROC curves (AUC) for each clinic. To account for the different age distributions between clinics, a method for age-standardizing the AUCs was developed in which the clinic-specific AUCs were standardized using weights for each embryo according to the relative frequency of the maternal age in the relevant clinic compared to the age distribution in a common reference population. Results: There was substantial variation in the clinic-specific AUCs with estimates ranging from 0.58 to 0.69 before standardization. The age-standardization of the AUCs reduced the between-clinic variance by 16%. Most notably, three of the clinics had quite similar AUCs after standardization, while the last clinic had a markedly lower AUC both with and without standardization. Conclusion: The method of using age-standardization of the AUCs that is proposed in this article mitigates some of the variability between clinics. This enables a comparison of clinic-specific AUCs where the difference in age distributions is accounted for.

KW - Artificial intelligence

KW - Embryo selection

KW - Model performance

KW - Time-lapse

KW - Fertilization in Vitro

KW - Humans

KW - Artificial Intelligence

KW - Blastocyst

KW - Retrospective Studies

KW - Time-Lapse Imaging

KW - Machine Learning

UR - http://www.scopus.com/inward/record.url?scp=85164161521&partnerID=8YFLogxK

U2 - 10.1007/s10815-023-02871-3

DO - 10.1007/s10815-023-02871-3

M3 - Journal article

C2 - 37423932

AN - SCOPUS:85164161521

SN - 1058-0468

VL - 40

SP - 2129

EP - 2137

JO - Journal of Assisted Reproduction and Genetics

JF - Journal of Assisted Reproduction and Genetics

IS - 9

ER -

Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning

Abstract

Keywords

Access to Document

OpenUrl availability

Other files and links

Fingerprint

Cite this