Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review

Standard

Procedure-specific assessment tool for flexible pharyngo-laryngoscopy : gathering validity evidence and setting pass–fail standards. / Melchiors, Jacob; Petersen, K.; Todsen, T.; Bohr, A.; Konge, Lars; von Buchwald, Christian.

I: European Archives of Oto-Rhino-Laryngology, Bind 275, Nr. 6, 06.2018, s. 1649-1655.

Publikation: Bidrag til tidsskrift/Konferencebidrag i tidsskrift /Bidrag til avisTidsskriftartikelForskningpeer review

Harvard

Melchiors, J, Petersen, K, Todsen, T, Bohr, A, Konge, L & von Buchwald, C 2018, 'Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards', European Archives of Oto-Rhino-Laryngology, bind 275, nr. 6, s. 1649-1655. https://doi.org/10.1007/s00405-018-4971-y

APA

Melchiors, J., Petersen, K., Todsen, T., Bohr, A., Konge, L., & von Buchwald, C. (2018). Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards. European Archives of Oto-Rhino-Laryngology, 275(6), 1649-1655. https://doi.org/10.1007/s00405-018-4971-y

CBE

Melchiors J, Petersen K, Todsen T, Bohr A, Konge L, von Buchwald C. 2018. Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards. European Archives of Oto-Rhino-Laryngology. 275(6):1649-1655. https://doi.org/10.1007/s00405-018-4971-y

MLA

Vancouver

Melchiors J, Petersen K, Todsen T, Bohr A, Konge L, von Buchwald C. Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards. European Archives of Oto-Rhino-Laryngology. 2018 jun;275(6):1649-1655. https://doi.org/10.1007/s00405-018-4971-y

Author

Melchiors, Jacob ; Petersen, K. ; Todsen, T. ; Bohr, A. ; Konge, Lars ; von Buchwald, Christian. / Procedure-specific assessment tool for flexible pharyngo-laryngoscopy : gathering validity evidence and setting pass–fail standards. I: European Archives of Oto-Rhino-Laryngology. 2018 ; Bind 275, Nr. 6. s. 1649-1655.

Bibtex

@article{d54d5214bd3c4ba587f2f578442b1696,
title = "Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards",
abstract = "Objective: The attainment of specific identifiable competencies is the primary measure of progress in the modern medical education system. The system, therefore, requires a method for accurately assessing competence to be feasible. Evidence of validity needs to be gathered before an assessment tool can be implemented in the training and assessment of physicians. This evidence of validity must according to the contemporary theory on validity be gathered from specific sources in a structured and rigorous manner. The flexible pharyngo-laryngoscopy (FPL) is central to the otorhinolaryngologist. We aim to evaluate the flexible pharyngo-laryngoscopy assessment tool (FLEXPAT) created in a previous study and to establish a pass–fail level for proficiency. Methods: Eighteen physicians with different levels of experience (novices, intermediates, and experienced) were recruited to the study. Each performed an FPL on two patients. These procedures were video recorded, blinded, and assessed by two specialists. The score was expressed as the percentage of a possible max score. Cronbach{\textquoteright}s α was used to analyze internal consistency of the data, and a generalizability analysis was performed. The scores of the three different groups were explored, and a pass–fail level was determined using the contrasting groups{\textquoteright} standard setting method. Results: Internal consistency was strong with a Cronbach{\textquoteright}s α of 0.86. We found a generalizability coefficient of 0.72 sufficient for moderate stakes assessment. We found a significant difference between the novice and experienced groups (p < 0.001) and strong correlation between experience and score (Pearson{\textquoteright}s r = 0.75). The pass/fail level was established at 72% of the maximum score. Applying this pass–fail level in the test population resulted in half of the intermediary group receiving a failing score. Discussion: We gathered validity evidence for the FLEXPAT according to the contemporary framework as described by Messick. Our results support a claim of validity and are comparable to other studies exploring clinical assessment tools. The high rate of physicians underperforming in the intermediary group demonstrates the need for continued educational intervention. Conclusion: Based on our work, we recommend the use of the FLEXPAT in clinical assessment of FPL and the application of a pass–fail level of 72% for proficiency.",
keywords = "Assessment tool, Flexible laryngoscopy, Mastery learning, Medical education, Technical skills, Validity",
author = "Jacob Melchiors and K. Petersen and T. Todsen and A. Bohr and Lars Konge and {von Buchwald}, Christian",
year = "2018",
month = jun,
doi = "10.1007/s00405-018-4971-y",
language = "English",
volume = "275",
pages = "1649--1655",
journal = "European Archives of Oto-Rhino-Laryngology",
issn = "0937-4477",
publisher = "Springer",
number = "6",

}

RIS

TY - JOUR

T1 - Procedure-specific assessment tool for flexible pharyngo-laryngoscopy

T2 - gathering validity evidence and setting pass–fail standards

AU - Melchiors, Jacob

AU - Petersen, K.

AU - Todsen, T.

AU - Bohr, A.

AU - Konge, Lars

AU - von Buchwald, Christian

PY - 2018/6

Y1 - 2018/6

N2 - Objective: The attainment of specific identifiable competencies is the primary measure of progress in the modern medical education system. The system, therefore, requires a method for accurately assessing competence to be feasible. Evidence of validity needs to be gathered before an assessment tool can be implemented in the training and assessment of physicians. This evidence of validity must according to the contemporary theory on validity be gathered from specific sources in a structured and rigorous manner. The flexible pharyngo-laryngoscopy (FPL) is central to the otorhinolaryngologist. We aim to evaluate the flexible pharyngo-laryngoscopy assessment tool (FLEXPAT) created in a previous study and to establish a pass–fail level for proficiency. Methods: Eighteen physicians with different levels of experience (novices, intermediates, and experienced) were recruited to the study. Each performed an FPL on two patients. These procedures were video recorded, blinded, and assessed by two specialists. The score was expressed as the percentage of a possible max score. Cronbach’s α was used to analyze internal consistency of the data, and a generalizability analysis was performed. The scores of the three different groups were explored, and a pass–fail level was determined using the contrasting groups’ standard setting method. Results: Internal consistency was strong with a Cronbach’s α of 0.86. We found a generalizability coefficient of 0.72 sufficient for moderate stakes assessment. We found a significant difference between the novice and experienced groups (p < 0.001) and strong correlation between experience and score (Pearson’s r = 0.75). The pass/fail level was established at 72% of the maximum score. Applying this pass–fail level in the test population resulted in half of the intermediary group receiving a failing score. Discussion: We gathered validity evidence for the FLEXPAT according to the contemporary framework as described by Messick. Our results support a claim of validity and are comparable to other studies exploring clinical assessment tools. The high rate of physicians underperforming in the intermediary group demonstrates the need for continued educational intervention. Conclusion: Based on our work, we recommend the use of the FLEXPAT in clinical assessment of FPL and the application of a pass–fail level of 72% for proficiency.

AB - Objective: The attainment of specific identifiable competencies is the primary measure of progress in the modern medical education system. The system, therefore, requires a method for accurately assessing competence to be feasible. Evidence of validity needs to be gathered before an assessment tool can be implemented in the training and assessment of physicians. This evidence of validity must according to the contemporary theory on validity be gathered from specific sources in a structured and rigorous manner. The flexible pharyngo-laryngoscopy (FPL) is central to the otorhinolaryngologist. We aim to evaluate the flexible pharyngo-laryngoscopy assessment tool (FLEXPAT) created in a previous study and to establish a pass–fail level for proficiency. Methods: Eighteen physicians with different levels of experience (novices, intermediates, and experienced) were recruited to the study. Each performed an FPL on two patients. These procedures were video recorded, blinded, and assessed by two specialists. The score was expressed as the percentage of a possible max score. Cronbach’s α was used to analyze internal consistency of the data, and a generalizability analysis was performed. The scores of the three different groups were explored, and a pass–fail level was determined using the contrasting groups’ standard setting method. Results: Internal consistency was strong with a Cronbach’s α of 0.86. We found a generalizability coefficient of 0.72 sufficient for moderate stakes assessment. We found a significant difference between the novice and experienced groups (p < 0.001) and strong correlation between experience and score (Pearson’s r = 0.75). The pass/fail level was established at 72% of the maximum score. Applying this pass–fail level in the test population resulted in half of the intermediary group receiving a failing score. Discussion: We gathered validity evidence for the FLEXPAT according to the contemporary framework as described by Messick. Our results support a claim of validity and are comparable to other studies exploring clinical assessment tools. The high rate of physicians underperforming in the intermediary group demonstrates the need for continued educational intervention. Conclusion: Based on our work, we recommend the use of the FLEXPAT in clinical assessment of FPL and the application of a pass–fail level of 72% for proficiency.

KW - Assessment tool

KW - Flexible laryngoscopy

KW - Mastery learning

KW - Medical education

KW - Technical skills

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=85045420577&partnerID=8YFLogxK

U2 - 10.1007/s00405-018-4971-y

DO - 10.1007/s00405-018-4971-y

M3 - Journal article

C2 - 29666918

AN - SCOPUS:85045420577

VL - 275

SP - 1649

EP - 1655

JO - European Archives of Oto-Rhino-Laryngology

JF - European Archives of Oto-Rhino-Laryngology

SN - 0937-4477

IS - 6

ER -