Aarhus University Seal / Aarhus Universitets segl

Dorte Guldbrand Nielsen

Clinical assessment of transthoracic echocardiography skills: a generalizability study

Research output: Contribution to journal/Conference contribution in journal/Contribution to newspaperJournal articleResearchpeer-review

Context: Transthoracic echocardiography (TTE) is a widely used cardiac imaging technique that all cardiologists should be able to perform competently. Traditionally, TTE competence has been assessed by unstructured observation or in test situations separated from daily clinical practice. An objective assessment instrument for TTE technical proficiency including a global rating score and a checklist score has previously been shown reliability and validity in a standardised setting.
Objectives: As clinical test situations typically have several sources of error giving rise to variance in scores, a more thorough examination of the generalizability of the test scores is needed.
Methods Nine physicians performed a TTE scan on the same three patients. Then, two raters rated all 27 TTE scans using the TTE technical assessment in a fully crossed generalizability study. Estimated variance components were calculated for both the global rating and checklist scores, and finally dependability (phi) coefficients were also calculated for both outcomes in a D-study.
Results For global rating scores, 66.6% of score variance can be ascribed to true differences in performance. For checklist scores this was 88.8%. This difference was primarily due to physician-rater interaction. Four random cases rated by one random rater resulted in a phi value of 0.81 for global ratings and 2 random cases rated by one random rater showed a phi value of 0.92 for checklist scores.
Discussion Using the TTE checklist as opposed to the TTE global rating score had the effect of minimising the largest source of error variance in test scores. Two random cases rated by one random rater using the TTE checklist are sufficiently reliable for a high stakes examination. As global rating is less time consuming it could be considered performing 4 global rating assessments in addition to the checklist assessments to account for both reliability and content validity of the assessment.
Original languageEnglish
JournalB M C Medical Education
Number of pages15
Publication statusPublished - 1 Feb 2015

See relations at Aarhus University Citationformats

ID: 84353260