Inter-observer and intra-observer agreement in drug-induced sedation endoscopy — a systematic approach

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review


Objective: To evaluate the feasibility of systematic observer training in drug-induced sedation endoscopy (DISE) interpretation. Methods: Fifty DISE videos were randomly selected from a group of 200 videos of cohort of patients with OSA and/or snoring. The videos were assessed blindly and independently using a modified VOTE classification by an expert observer and by two novice observers starting their training. A systematic approach was initiated. Clusters of 10 videos were scored individually by each observer and then re-evaluated as the expert observer clarified the decision-making in plenum. Kappa coefficient (κ) was calculated as a measure of agreement. Results: The intra-observer variation for the total agreement and kappa values for the expert observer ranged between moderate to substantial agreement in VOTE classification, whereas the novices varied between “less than chance agreement” to “moderate agreement.” The inter-observer variation showed increased agreement and kappa values from day 1 to day 2 for both novice observers except at the velum level for observer C. The total agreement and kappa values for each site also improved compared to results of day 1, except at velum for observer C. The velum site seemed to be more difficult to evaluate. The learning curve varied during the study course for each site of the upper airways. Conclusion: This study shows that systematic intensive training is feasible, although expert level is not acquired after 100 evaluations. The learning curve for the expert observer showed “moderate to substantial agreement” but differed between the trainees. Level of evidence: 2.

TidsskriftEgyptian Journal of Otolaryngology
StatusUdgivet - apr. 2022

Bibliografisk note

Funding Information:
This study has been supervised by a certified statistician at the Section of Biostatistics at the Department of Public Health, University of Aarhus, Denmark.

Publisher Copyright:
© 2022, The Author(s).

Se relationer på Aarhus Universitet Citationsformater

ID: 272532792