Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation

Kareem A. Wahid; Onur Sahin; Suprateek Kundu; Diana Lin; Anthony Alanis; Salik Tehami; Serageldin Kamel; Simon Duke; Michael V. Sherer; Mathis Rasmussen; Stine Korreman; David Fuentes; Michael Cislo; Benjamin E. Nelms; John P. Christodouleas; James D. Murphy; Abdallah S.R. Mohamed; Renjie He; Mohammed A. Naser; Erin F. Gillespie; Clifton D. Fuller

doi:10.1200/CCI.23.00174

Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation

Kareem A. Wahid, Onur Sahin, Suprateek Kundu, Diana Lin, Anthony Alanis, Salik Tehami, Serageldin Kamel, Simon Duke, Michael V. Sherer, Mathis Rasmussen, Stine Korreman, David Fuentes, Michael Cislo, Benjamin E. Nelms, John P. Christodouleas, James D. Murphy, Abdallah S.R. Mohamed, Renjie He, Mohammed A. Naser, Erin F. GillespieClifton D. Fuller

Department of Clinical Medicine - The Department of Oncology

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

Abstract

PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.

Original language	English
Article number	e2300174
Journal	JCO Clinical Cancer Informatics
Volume	8
Pages (from-to)	e2300174
ISSN	2473-4276
DOIs	https://doi.org/10.1200/CCI.23.00174
Publication status	Published - 1 Jun 2024

Access to Document

10.1200/CCI.23.00174

Cite this

Wahid, K. A., Sahin, O., Kundu, S., Lin, D., Alanis, A., Tehami, S., Kamel, S., Duke, S., Sherer, M. V., Rasmussen, M., Korreman, S., Fuentes, D., Cislo, M., Nelms, B. E., Christodouleas, J. P., Murphy, J. D., Mohamed, A. S. R., He, R., Naser, M. A., ... Fuller, C. D. (2024). Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation. JCO Clinical Cancer Informatics, 8, e2300174. Article e2300174. https://doi.org/10.1200/CCI.23.00174

@article{1e8676c41d9f47e38999be56d3fc161b,

title = "Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation",

abstract = "PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.",

author = "Wahid, {Kareem A.} and Onur Sahin and Suprateek Kundu and Diana Lin and Anthony Alanis and Salik Tehami and Serageldin Kamel and Simon Duke and Sherer, {Michael V.} and Mathis Rasmussen and Stine Korreman and David Fuentes and Michael Cislo and Nelms, {Benjamin E.} and Christodouleas, {John P.} and Murphy, {James D.} and Mohamed, {Abdallah S.R.} and Renjie He and Naser, {Mohammed A.} and Gillespie, {Erin F.} and Fuller, {Clifton D.}",

year = "2024",

month = jun,

day = "1",

doi = "10.1200/CCI.23.00174",

language = "English",

volume = "8",

pages = "e2300174",

journal = "JCO Clinical Cancer Informatics",

issn = "2473-4276",

publisher = "Lippincott Williams and Wilkins",

}

Wahid, KA, Sahin, O, Kundu, S, Lin, D, Alanis, A, Tehami, S, Kamel, S, Duke, S, Sherer, MV, Rasmussen, M , Korreman, S, Fuentes, D, Cislo, M, Nelms, BE, Christodouleas, JP, Murphy, JD, Mohamed, ASR, He, R, Naser, MA, Gillespie, EF & Fuller, CD 2024, 'Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation', JCO Clinical Cancer Informatics, vol. 8, e2300174, pp. e2300174. https://doi.org/10.1200/CCI.23.00174

Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation. / Wahid, Kareem A.; Sahin, Onur; Kundu, Suprateek et al.
In: JCO Clinical Cancer Informatics, Vol. 8, e2300174, 01.06.2024, p. e2300174.

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaper › Journal article › Research › peer-review

TY - JOUR

T1 - Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks

T2 - Insights From a Crowd-Sourced Challenge Using Bayesian Estimation

AU - Wahid, Kareem A.

AU - Sahin, Onur

AU - Kundu, Suprateek

AU - Lin, Diana

AU - Alanis, Anthony

AU - Tehami, Salik

AU - Kamel, Serageldin

AU - Duke, Simon

AU - Sherer, Michael V.

AU - Rasmussen, Mathis

AU - Korreman, Stine

AU - Fuentes, David

AU - Cislo, Michael

AU - Nelms, Benjamin E.

AU - Christodouleas, John P.

AU - Murphy, James D.

AU - Mohamed, Abdallah S.R.

AU - He, Renjie

AU - Naser, Mohammed A.

AU - Gillespie, Erin F.

AU - Fuller, Clifton D.

PY - 2024/6/1

Y1 - 2024/6/1

N2 - PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.

AB - PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.

UR - http://www.scopus.com/inward/record.url?scp=85196137760&partnerID=8YFLogxK

U2 - 10.1200/CCI.23.00174

DO - 10.1200/CCI.23.00174

M3 - Journal article

C2 - 38870441

AN - SCOPUS:85196137760

SN - 2473-4276

VL - 8

SP - e2300174

JO - JCO Clinical Cancer Informatics

JF - JCO Clinical Cancer Informatics

M1 - e2300174

ER -

Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation

Abstract

Access to Document

Other files and links

Fingerprint

Cite this